Improve compression of Arrays of Integers (High compression mode) #3895
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
this works great for 32-bit arrays, notably the synthetic ones, with extreme regularity, unfortunately, it's not universal, and in some cases, it's a loss. Crucially, on average, it's a loss on silesia. The most negatively impacted file is x-ray. It deserves an investigation before suggesting it as an evolution.
store stretches as intermediate solution instead of sequences. makes it possible to link a solution to a predecessor.
helps when litlen==1 is cheaper than litlen==0 works great on pathological arr[u32] examples but doesn't generalize well on other files. silesia/x-ray is amoung the most negatively affected ones.
While it's not always strictly a win, it's a win for files that see a noticeably compression ratio increase, while it's a very small noise for other files. Downside is, this patch is less efficient for 32-bit arrays of integer than the previous patch which was introducing losses for other files, but it's still a net improvement on this scenario.
good news: there are only improvements
by introducing ZSTD_OPT_SIZE
which can be redirected in Linux kernel mode
|
So there is still a reported problem ("use-after-poison") in one of the tests ( edit : now fixed, but could probably have been easier. |
terrelln
approved these changes
Feb 5, 2024
| lastSequence.mlen = maxML; | ||
| lastSequence.off = maxOffBase; | ||
| DEBUGLOG(6, "large match (%u>%u), immediate encoding", | ||
| lastStretch.litlen = 0; |
There was a problem hiding this comment.
I don't understand this change.
Edit: After reading the code I understand. It would've been helpful to see a description of the change that this diff made in the summary, because it was non-obvious until I finished reading the code.
| * - So far, @opt stored stretches, aka a match followed by literals | ||
| * - Now, it will store sequences, aka literals followed by a match | ||
| */ | ||
| { U32 const storeEnd = cur + 2; |
There was a problem hiding this comment.
Why +2? Also update the docs which say that storeEnd == cur+1.
There was a problem hiding this comment.
Note below, if lastStretch.litlen > 0, i.e. there are some literals after the last match,
then we start to write into opt[storeStart] with storeStart == storeEnd-1,
therefore overwriting the content of opt[storeStart]. And if storeEnd==cur+1, then storeStart would be ==cur.
However, opt[cur] is where the next stretch is stored, and we haven't read it yet, since we are still dealing with lastStretch. So we would be overwriting a component of our next sequence.
A +2 avoids this overwriting.
This is fine because ZSTD_OPT_SIZE == ZSTD_OPT_NUM + 2, so we have enough margin, even when the array is entirely filled. This is checked in the following assert : assert(storeEnd < ZSTD_OPT_SIZE);.
Comment on lines
+1209
to
+1233
| if ( (optLevel >= 1) /* additional check only for higher modes */ | ||
| && (prevMatch.litlen == 0) /* replace a match */ | ||
| && (LL_INCPRICE(1) < 0) /* ll1 is cheaper than ll0 */ | ||
| ) { | ||
| /* check next position, in case it would be cheaper */ | ||
| int with1literal = prevMatch.price + LIT_PRICE(ip+cur) + LL_INCPRICE(1); | ||
| int withMoreLiterals = price + LIT_PRICE(ip+cur) + LL_INCPRICE(litlen+1); | ||
| DEBUGLOG(7, "then at next rPos %u : match+1lit %.2f vs %ulits %.2f", | ||
| cur+1, ZSTD_fCost(with1literal), litlen+1, ZSTD_fCost(withMoreLiterals)); | ||
| if ( (with1literal < withMoreLiterals) | ||
| && (with1literal < opt[cur+1].price) ) { | ||
| /* update offset history - before it disappears */ | ||
| U32 const prev = cur - prevMatch.mlen; | ||
| repcodes_t const newReps = ZSTD_newRep(opt[prev].rep, prevMatch.off, opt[prev].litlen==0); | ||
| assert(cur >= prevMatch.mlen); | ||
| DEBUGLOG(7, "==> match+1lit is cheaper (%.2f < %.2f) (hist:%u,%u,%u) !", | ||
| ZSTD_fCost(with1literal), ZSTD_fCost(withMoreLiterals), | ||
| newReps.rep[0], newReps.rep[1], newReps.rep[2] ); | ||
| opt[cur+1] = prevMatch; /* mlen & offbase */ | ||
| ZSTD_memcpy(opt[cur+1].rep, &newReps, sizeof(repcodes_t)); | ||
| opt[cur+1].litlen = 1; | ||
| opt[cur+1].price = with1literal; | ||
| if (last_pos < cur+1) last_pos = cur+1; | ||
| } | ||
| } |
There was a problem hiding this comment.
Besides the change to litlength, which is to facilitate this change, this section is the major functional change of this diff, correct?
There was a problem hiding this comment.
Besides the change to
litlength, which is to facilitate this change, this section is the major functional change of this diff, correct?
Yes
as suggested by @terrelln, to make the code of the optimal parser a bit more understandable.
This was referenced Feb 6, 2024
Merged
anakinxc
referenced
this pull request
in secretflow/spu
Mar 27, 2024
[](https://renovatebot.com) This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [com_github_facebook_zstd](https://togithub.com/facebook/zstd) | http_archive | patch | `v1.5.5` -> `v1.5.6` | --- ### Release Notes <details> <summary>facebook/zstd (com_github_facebook_zstd)</summary> ### [`v1.5.6`](https://togithub.com/facebook/zstd/releases/tag/v1.5.6): Zstandard v1.5.6 - Chrome Edition [Compare Source](https://togithub.com/facebook/zstd/compare/v1.5.5-kernel...v1.5.6) This release highlights the deployment of Google [Chrome 123](https://developer.chrome.com/blog/new-in-chrome-123), introducing `zstd-encoding` for Web traffic, introduced as a preferable option for compression of dynamic contents. With limited web server support for `zstd-encoding` due to its novelty, we are launching an updated Zstandard version to facilitate broader adoption. ##### New stable parameter `ZSTD_c_targetCBlockSize` Using `zstd` compression for large documents over the Internet, data is segmented into smaller blocks of up to 128 KB, for incremental updates. This is crucial for applications like Chrome that process parts of documents as they arrive. However, on slow or congested networks, there can be some brief unresponsiveness in the middle of a block transmission, delaying update. To mitigate such scenarios, `libzstd` introduces the new parameter `ZSTD_c_targetCBlockSize`, enabling the division of blocks into even smaller segments to enhance initial byte delivery speed. Activating this feature incurs a cost, both runtime (equivalent to -2% speed at level 8) and a slight compression efficiency decrease (<0.1%), but offers some interesting latency reduction, notably beneficial in areas with less powerful network infrastructure. ##### Granular binary size selection `libzstd` provides build customization, including options to compile only the compression or decompression modules, minimizing binary size. Enhanced in `v1.5.6` ([source](https://togithub.com/facebook/zstd/tree/dev/lib#modular-build)), it now allows for even finer control by enabling selective inclusion or exclusion of specific components within these modules. This advancement aids applications needing precise binary size management. ##### Miscellaneous Enhancements This release includes various minor enhancements and bug fixes to enhance user experience. Key updates include an expanded list of recognized compressed file suffixes for the `--exclude-compressed` flag, improving efficiency by skipping presumed incompressible content. Furthermore, compatibility has been broadened to include additional chipsets (`sparc64`, `ARM64EC`, `risc-v`) and operating systems (`QNX`, `AIX`, `Solaris`, `HP-UX`). #### Change Log api: Promote `ZSTD_c_targetCBlockSize` to Stable API by [@​felixhandte](https://togithub.com/felixhandte) api: new experimental `ZSTD_d_maxBlockSize` parameter, to reduce streaming decompression memory, by [@​terrelln](https://togithub.com/terrelln) perf: improve performance of param `ZSTD_c_targetCBlockSize`, by [@​Cyan4973](https://togithub.com/Cyan4973) perf: improved compression of arrays of integers at high compression, by [@​Cyan4973](https://togithub.com/Cyan4973) lib: reduce binary size with selective built-time exclusion, by [@​felixhandte](https://togithub.com/felixhandte) lib: improved huffman speed on small data and linux kernel, by [@​terrelln](https://togithub.com/terrelln) lib: accept dictionaries with partial literal tables, by [@​terrelln](https://togithub.com/terrelln) lib: fix CCtx size estimation with external sequence producer, by [@​embg](https://togithub.com/embg) lib: fix corner case decoder behaviors, by [@​Cyan4973](https://togithub.com/Cyan4973) and [@​aimuz](https://togithub.com/aimuz) lib: fix zdict prototype mismatch in static_only mode, by [@​ldv-alt](https://togithub.com/ldv-alt) lib: fix several bugs in magicless-format decoding, by [@​embg](https://togithub.com/embg) cli: add common compressed file types to `--exclude-compressed` by [@​daniellerozenblit](https://togithub.com/daniellerozenblit) (requested by [@​dcog989](https://togithub.com/dcog989)) cli: fix mixing `-c` and `-o` commands with `--rm`, by [@​Cyan4973](https://togithub.com/Cyan4973) cli: fix erroneous exclusion of hidden files with `--output-dir-mirror` by [@​felixhandte](https://togithub.com/felixhandte) cli: improved time accuracy on BSD, by [@​felixhandte](https://togithub.com/felixhandte) cli: better errors on argument parsing, by [@​KapJI](https://togithub.com/KapJI) tests: better compatibility with older versions of `grep`, by [@​Cyan4973](https://togithub.com/Cyan4973) tests: lorem ipsum generator as default content generator, by [@​Cyan4973](https://togithub.com/Cyan4973) build: cmake improvements by [@​terrelln](https://togithub.com/terrelln), [@​sighingnow](https://togithub.com/sighingnow), [@​gjasny](https://togithub.com/gjasny), [@​JohanMabille](https://togithub.com/JohanMabille), [@​Saverio976](https://togithub.com/Saverio976), [@​gruenich](https://togithub.com/gruenich), [@​teo-tsirpanis](https://togithub.com/teo-tsirpanis) build: bazel support, by [@​jondo2010](https://togithub.com/jondo2010) build: fix cross-compiling for AArch64 with lld by [@​jcelerier](https://togithub.com/jcelerier) build: fix Apple platform compatibility, by [@​nidhijaju](https://togithub.com/nidhijaju) build: fix Visual 2012 and lower compatibility, by [@​Cyan4973](https://togithub.com/Cyan4973) build: improve win32 support, by [@​DimitriPapadopoulos](https://togithub.com/DimitriPapadopoulos) build: better C90 compliance for zlibWrapper, by [@​emaste](https://togithub.com/emaste) port: make: fat binaries on macos, by [@​mredig](https://togithub.com/mredig) port: ARM64EC compatibility for Windows, by [@​dunhor](https://togithub.com/dunhor) port: QNX support by [@​klausholstjacobsen](https://togithub.com/klausholstjacobsen) port: MSYS2 and Cygwin makefile installation and test support, by [@​QBos07](https://togithub.com/QBos07) port: risc-v support validation in CI, by [@​Cyan4973](https://togithub.com/Cyan4973) port: sparc64 support validation in CI, by [@​Cyan4973](https://togithub.com/Cyan4973) port: AIX compatibility, by [@​likema](https://togithub.com/likema) port: HP-UX compatibility, by [@​likema](https://togithub.com/likema) doc: Improved specification accuracy, by [@​elasota](https://togithub.com/elasota) bug: Fix and deprecate ZSTD_generateSequences ([#​3981](https://togithub.com/facebook/zstd/issues/3981)), by [@​terrelln](https://togithub.com/terrelln) #### Full change list (auto-generated) - Add win32 to windows-artifacts.yml by [@​Kim-SSi](https://togithub.com/Kim-SSi) in [https://github.com/facebook/zstd/pull/3600](https://togithub.com/facebook/zstd/pull/3600) - Fix mmap-dict help output by [@​daniellerozenblit](https://togithub.com/daniellerozenblit) in [https://github.com/facebook/zstd/pull/3601](https://togithub.com/facebook/zstd/pull/3601) - \[oss-fuzz] Fix simple_round_trip fuzzer with overlapping decompression by [@​terrelln](https://togithub.com/terrelln) in [https://github.com/facebook/zstd/pull/3612](https://togithub.com/facebook/zstd/pull/3612) - Reduce streaming decompression memory by (128KB - blockSizeMax) by [@​terrelln](https://togithub.com/terrelln) in [https://github.com/facebook/zstd/pull/3616](https://togithub.com/facebook/zstd/pull/3616) - removed travis & appveyor scripts by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3621](https://togithub.com/facebook/zstd/pull/3621) - Add ZSTD_d_maxBlockSize parameter by [@​terrelln](https://togithub.com/terrelln) in [https://github.com/facebook/zstd/pull/3617](https://togithub.com/facebook/zstd/pull/3617) - \[doc] add decoder errata paragraph by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3620](https://togithub.com/facebook/zstd/pull/3620) - add makefile entry to build fat binary on macos by [@​mredig](https://togithub.com/mredig) in [https://github.com/facebook/zstd/pull/3614](https://togithub.com/facebook/zstd/pull/3614) - Disable unused variable warning in msan configurations by [@​danlark1](https://togithub.com/danlark1) in [https://github.com/facebook/zstd/pull/3624](https://togithub.com/facebook/zstd/pull/3624) [https://github.com/facebook/zstd/pull/3634](https://togithub.com/facebook/zstd/pull/3634)3634 - Allow Build-Time Exclusion of Individual Compression Strategies by [@​felixhandte](https://togithub.com/felixhandte) in [https://github.com/facebook/zstd/pull/3623](https://togithub.com/facebook/zstd/pull/3623) - Get zstd working with ARM64EC on Windows by [@​dunhor](https://togithub.com/dunhor) in [https://github.com/facebook/zstd/pull/3636](https://togithub.com/facebook/zstd/pull/3636) - minor : update streaming_compression example by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3631](https://togithub.com/facebook/zstd/pull/3631) - Fix UBSAN issue (zero addition to NULL) by [@​terrelln](https://togithub.com/terrelln) in [https://github.com/facebook/zstd/pull/3658](https://togithub.com/facebook/zstd/pull/3658) - Add options in Makefile to cmake by [@​sighingnow](https://togithub.com/sighingnow) in [https://github.com/facebook/zstd/pull/3657](https://togithub.com/facebook/zstd/pull/3657) - fix a minor inefficiency in compress_superblock by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3668](https://togithub.com/facebook/zstd/pull/3668) - Fixed a bug in the educational decoder by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3659](https://togithub.com/facebook/zstd/pull/3659) - changed LLU suffix into ULL for Visual 2012 and lower by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3664](https://togithub.com/facebook/zstd/pull/3664) - fixed decoder behavior when nbSeqs==0 is encoded using 2 bytes by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3669](https://togithub.com/facebook/zstd/pull/3669) - detect extraneous bytes in the Sequences section by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3674](https://togithub.com/facebook/zstd/pull/3674) - Bitstream produces only zeroes after an overflow event by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3676](https://togithub.com/facebook/zstd/pull/3676) - Update FreeBSD CI images to latest supported releases by [@​emaste](https://togithub.com/emaste) in [https://github.com/facebook/zstd/pull/3684](https://togithub.com/facebook/zstd/pull/3684) - Clean up a false error message in the LDM debug log by [@​embg](https://togithub.com/embg) in [https://github.com/facebook/zstd/pull/3686](https://togithub.com/facebook/zstd/pull/3686) - Hide ASM symbols on Apple platforms by [@​nidhijaju](https://togithub.com/nidhijaju) in [https://github.com/facebook/zstd/pull/3688](https://togithub.com/facebook/zstd/pull/3688) - Changed the decoding loop to detect more invalid cases of corruption sooner by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3677](https://togithub.com/facebook/zstd/pull/3677) - Fix Intel Xcode builds with assembly by [@​gjasny](https://togithub.com/gjasny) in [https://github.com/facebook/zstd/pull/3665](https://togithub.com/facebook/zstd/pull/3665) - Save one byte on the frame epilogue by [@​Coder-256](https://togithub.com/Coder-256) in [https://github.com/facebook/zstd/pull/3700](https://togithub.com/facebook/zstd/pull/3700) - Update fileio.c: fix build failure with enabled LTO by [@​LocutusOfBorg](https://togithub.com/LocutusOfBorg) in [https://github.com/facebook/zstd/pull/3695](https://togithub.com/facebook/zstd/pull/3695) - fileio_asyncio: handle malloc fails in AIO_ReadPool_create by [@​void0red](https://togithub.com/void0red) in [https://github.com/facebook/zstd/pull/3704](https://togithub.com/facebook/zstd/pull/3704) - Fix typographical error in README.md by [@​nikohoffren](https://togithub.com/nikohoffren) in [https://github.com/facebook/zstd/pull/3701](https://togithub.com/facebook/zstd/pull/3701) - Fixed typo by [@​alexsifivetw](https://togithub.com/alexsifivetw) in [https://github.com/facebook/zstd/pull/3712](https://togithub.com/facebook/zstd/pull/3712) - Improve dual license wording in README by [@​terrelln](https://togithub.com/terrelln) in [https://github.com/facebook/zstd/pull/3718](https://togithub.com/facebook/zstd/pull/3718) - Unpoison Workspace Memory Before Custom-Free by [@​felixhandte](https://togithub.com/felixhandte) in [https://github.com/facebook/zstd/pull/3725](https://togithub.com/facebook/zstd/pull/3725) - added ZSTD_decompressDCtx() benchmark option to fullbench by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3726](https://togithub.com/facebook/zstd/pull/3726) - No longer reject dictionaries with literals maxSymbolValue < 255 by [@​terrelln](https://togithub.com/terrelln) in [https://github.com/facebook/zstd/pull/3731](https://togithub.com/facebook/zstd/pull/3731) - fix: ZSTD_BUILD_DECOMPRESSION message by [@​0o001](https://togithub.com/0o001) in [https://github.com/facebook/zstd/pull/3728](https://togithub.com/facebook/zstd/pull/3728) - Updated Makefiles for full MSYS2 and Cygwin installation and testing … by [@​QBos07](https://togithub.com/QBos07) in [https://github.com/facebook/zstd/pull/3720](https://togithub.com/facebook/zstd/pull/3720) - Work around nullptr-with-nonzero-offset warning by [@​terrelln](https://togithub.com/terrelln) in [https://github.com/facebook/zstd/pull/3738](https://togithub.com/facebook/zstd/pull/3738) - Fix & refactor Huffman repeat tables for dictionaries by [@​terrelln](https://togithub.com/terrelln) in [https://github.com/facebook/zstd/pull/3737](https://togithub.com/facebook/zstd/pull/3737) - zdictlib: fix prototype mismatch by [@​ldv-alt](https://togithub.com/ldv-alt) in [https://github.com/facebook/zstd/pull/3733](https://togithub.com/facebook/zstd/pull/3733) - Fixed zstd cmake shared build on windows by [@​JohanMabille](https://togithub.com/JohanMabille) in [https://github.com/facebook/zstd/pull/3739](https://togithub.com/facebook/zstd/pull/3739) - Added qnx in the posix test section of platform.h by [@​klausholstjacobsen](https://togithub.com/klausholstjacobsen) in [https://github.com/facebook/zstd/pull/3745](https://togithub.com/facebook/zstd/pull/3745) - added some documentation on ZSTD_estimate\*Size() variants by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3755](https://togithub.com/facebook/zstd/pull/3755) - Improve macro guards for ZSTD_assertValidSequence by [@​terrelln](https://togithub.com/terrelln) in [https://github.com/facebook/zstd/pull/3770](https://togithub.com/facebook/zstd/pull/3770) - Stop suppressing pointer-overflow UBSAN errors by [@​terrelln](https://togithub.com/terrelln) in [https://github.com/facebook/zstd/pull/3776](https://togithub.com/facebook/zstd/pull/3776) - fix x32 tests on Github CI by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3777](https://togithub.com/facebook/zstd/pull/3777) - Fix new typos found by codespell by [@​DimitriPapadopoulos](https://togithub.com/DimitriPapadopoulos) in [https://github.com/facebook/zstd/pull/3771](https://togithub.com/facebook/zstd/pull/3771) - Do not test WIN32, instead test \_WIN32 by [@​DimitriPapadopoulos](https://togithub.com/DimitriPapadopoulos) in [https://github.com/facebook/zstd/pull/3772](https://togithub.com/facebook/zstd/pull/3772) - Fix a very small formatting typo in the lib/README.md file by [@​dloidolt](https://togithub.com/dloidolt) in [https://github.com/facebook/zstd/pull/3763](https://togithub.com/facebook/zstd/pull/3763) - Fix pzstd Makefile to allow setting `DESTDIR` and `BINDIR` separately by [@​paulmenzel](https://togithub.com/paulmenzel) in [https://github.com/facebook/zstd/pull/3752](https://togithub.com/facebook/zstd/pull/3752) - Remove FlexArray pattern from ZSTDMT by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3786](https://togithub.com/facebook/zstd/pull/3786) - solving flexArray issue [#​3785](https://togithub.com/facebook/zstd/issues/3785) in fse by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3789](https://togithub.com/facebook/zstd/pull/3789) - Add doc on how to use it with cmake FetchContent by [@​Saverio976](https://togithub.com/Saverio976) in [https://github.com/facebook/zstd/pull/3795](https://togithub.com/facebook/zstd/pull/3795) - Correct FSE probability bit consumption in specification by [@​elasota](https://togithub.com/elasota) in [https://github.com/facebook/zstd/pull/3806](https://togithub.com/facebook/zstd/pull/3806) - Add Bazel module instructions to README.md by [@​jondo2010](https://togithub.com/jondo2010) in [https://github.com/facebook/zstd/pull/3812](https://togithub.com/facebook/zstd/pull/3812) - Clarify that a stream containing too many Huffman weights is invalid by [@​elasota](https://togithub.com/elasota) in [https://github.com/facebook/zstd/pull/3813](https://togithub.com/facebook/zstd/pull/3813) - \[cmake] Require CMake version 3.5 or newer by [@​gruenich](https://togithub.com/gruenich) in [https://github.com/facebook/zstd/pull/3807](https://togithub.com/facebook/zstd/pull/3807) - Three fixes for the Linux kernel by [@​terrelln](https://togithub.com/terrelln) in [https://github.com/facebook/zstd/pull/3822](https://togithub.com/facebook/zstd/pull/3822) - \[huf] Improve fast huffman decoding speed in linux kernel by [@​terrelln](https://togithub.com/terrelln) in [https://github.com/facebook/zstd/pull/3826](https://togithub.com/facebook/zstd/pull/3826) - \[huf] Improve fast C & ASM performance on small data by [@​terrelln](https://togithub.com/terrelln) in [https://github.com/facebook/zstd/pull/3827](https://togithub.com/facebook/zstd/pull/3827) - update xxhash library to v0.8.2 by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3820](https://togithub.com/facebook/zstd/pull/3820) - Modernize macros to use `do { } while (0)` by [@​terrelln](https://togithub.com/terrelln) in [https://github.com/facebook/zstd/pull/3831](https://togithub.com/facebook/zstd/pull/3831) - Clarify that the presence of weight value 1 is required, and a lone implied 1 weight is invalid by [@​elasota](https://togithub.com/elasota) in [https://github.com/facebook/zstd/pull/3814](https://togithub.com/facebook/zstd/pull/3814) - Move offload API params into ZSTD_CCtx_params by [@​embg](https://togithub.com/embg) in [https://github.com/facebook/zstd/pull/3839](https://togithub.com/facebook/zstd/pull/3839) - Update FreeBSD CI: drop 12.4 (nearly EOL) by [@​emaste](https://togithub.com/emaste) in [https://github.com/facebook/zstd/pull/3845](https://togithub.com/facebook/zstd/pull/3845) - Make offload API compatible with static CCtx by [@​embg](https://togithub.com/embg) in [https://github.com/facebook/zstd/pull/3854](https://togithub.com/facebook/zstd/pull/3854) - zlibWrapper: convert to C89 / ANSI C by [@​emaste](https://togithub.com/emaste) in [https://github.com/facebook/zstd/pull/3846](https://togithub.com/facebook/zstd/pull/3846) - Fix a nullptr dereference in ZSTD_createCDict_advanced2() by [@​michoecho](https://togithub.com/michoecho) in [https://github.com/facebook/zstd/pull/3847](https://togithub.com/facebook/zstd/pull/3847) - Cirrus-CI: Add FreeBSD 14 by [@​emaste](https://togithub.com/emaste) in [https://github.com/facebook/zstd/pull/3855](https://togithub.com/facebook/zstd/pull/3855) - CI: meson: use builtin handling for MSVC by [@​eli-schwartz](https://togithub.com/eli-schwartz) in [https://github.com/facebook/zstd/pull/3858](https://togithub.com/facebook/zstd/pull/3858) - cli: better errors on argument parsing by [@​KapJI](https://togithub.com/KapJI) in [https://github.com/facebook/zstd/pull/3850](https://togithub.com/facebook/zstd/pull/3850) - Clarify that probability tables must not contain non-zero probabilities for invalid values by [@​elasota](https://togithub.com/elasota) in [https://github.com/facebook/zstd/pull/3817](https://togithub.com/facebook/zstd/pull/3817) - \[x-compile] Fix cross-compiling for AArch64 with lld by [@​jcelerier](https://togithub.com/jcelerier) in [https://github.com/facebook/zstd/pull/3760](https://togithub.com/facebook/zstd/pull/3760) - playTests.sh does no longer needs grep -E by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3865](https://togithub.com/facebook/zstd/pull/3865) - minor: playTests.sh more compatible with older versions of grep by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3877](https://togithub.com/facebook/zstd/pull/3877) - disable Intel CET Compatibility tests by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3884](https://togithub.com/facebook/zstd/pull/3884) - improve cmake test by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3883](https://togithub.com/facebook/zstd/pull/3883) - add sparc64 compilation test by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3886](https://togithub.com/facebook/zstd/pull/3886) - add a lorem ipsum generator by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3890](https://togithub.com/facebook/zstd/pull/3890) - Update Dependency in Intel CET Test; Re-Enable Test by [@​felixhandte](https://togithub.com/felixhandte) in [https://github.com/facebook/zstd/pull/3893](https://togithub.com/facebook/zstd/pull/3893) - Improve compression of Arrays of Integers (High compression mode) by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3895](https://togithub.com/facebook/zstd/pull/3895) - \[Zstd] Less verbose log for patch mode. by [@​sandreenko](https://togithub.com/sandreenko) in [https://github.com/facebook/zstd/pull/3899](https://togithub.com/facebook/zstd/pull/3899) - fix [`5921623`](https://togithub.com/facebook/zstd/commit/5921623844651008) by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3900](https://togithub.com/facebook/zstd/pull/3900) - Fix fuzz issue [`5131069`](https://togithub.com/facebook/zstd/commit/5131069967892480) by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3902](https://togithub.com/facebook/zstd/pull/3902) - Advertise Availability of Security Vulnerability Notifications by [@​felixhandte](https://togithub.com/felixhandte) in [https://github.com/facebook/zstd/pull/3909](https://togithub.com/facebook/zstd/pull/3909) - updated setup-msys2 to v2.22.0 by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3914](https://togithub.com/facebook/zstd/pull/3914) - Lorem Ipsum generator update by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3913](https://togithub.com/facebook/zstd/pull/3913) - Reduce scope of variables by [@​gruenich](https://togithub.com/gruenich) in [https://github.com/facebook/zstd/pull/3903](https://togithub.com/facebook/zstd/pull/3903) - Improve speed of ZSTD_c_targetCBlockSize by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3915](https://togithub.com/facebook/zstd/pull/3915) - More regular block sizes with `targetCBlockSize` by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3917](https://togithub.com/facebook/zstd/pull/3917) - removed sprintf usage from zstdcli.c by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3916](https://togithub.com/facebook/zstd/pull/3916) - Export a `zstd::libzstd` CMake target if only static or dynamic linkage is specified. by [@​teo-tsirpanis](https://togithub.com/teo-tsirpanis) in [https://github.com/facebook/zstd/pull/3811](https://togithub.com/facebook/zstd/pull/3811) - fix version of actions/checkout by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3926](https://togithub.com/facebook/zstd/pull/3926) - minor Makefile refactoring by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3753](https://togithub.com/facebook/zstd/pull/3753) - lib/decompress: check for reserved bit corruption in zstd by [@​aimuz](https://togithub.com/aimuz) in [https://github.com/facebook/zstd/pull/3840](https://togithub.com/facebook/zstd/pull/3840) - Fix state table formatting by [@​elasota](https://togithub.com/elasota) in [https://github.com/facebook/zstd/pull/3816](https://togithub.com/facebook/zstd/pull/3816) - Specify offset 0 as invalid and specify required fixup behavior by [@​elasota](https://togithub.com/elasota) in [https://github.com/facebook/zstd/pull/3824](https://togithub.com/facebook/zstd/pull/3824) - update -V documentation by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3928](https://togithub.com/facebook/zstd/pull/3928) - fix LLU->ULL by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3929](https://togithub.com/facebook/zstd/pull/3929) - Fix building xxhash on AIX 5.1 by [@​likema](https://togithub.com/likema) in [https://github.com/facebook/zstd/pull/3860](https://togithub.com/facebook/zstd/pull/3860) - Fix building on HP-UX 11.11 PA-RISC by [@​likema](https://togithub.com/likema) in [https://github.com/facebook/zstd/pull/3862](https://togithub.com/facebook/zstd/pull/3862) - Fix AsyncIO reading seed queueing by [@​yoniko](https://togithub.com/yoniko) in [https://github.com/facebook/zstd/pull/3940](https://togithub.com/facebook/zstd/pull/3940) - Use ZSTD_LEGACY_SUPPORT=5 in "make test" by [@​embg](https://togithub.com/embg) in [https://github.com/facebook/zstd/pull/3943](https://togithub.com/facebook/zstd/pull/3943) - Pin sanitizer CI jobs to ubuntu-20.04 by [@​embg](https://togithub.com/embg) in [https://github.com/facebook/zstd/pull/3945](https://togithub.com/facebook/zstd/pull/3945) - chore: fix some typos by [@​acceptacross](https://togithub.com/acceptacross) in [https://github.com/facebook/zstd/pull/3949](https://togithub.com/facebook/zstd/pull/3949) - new method to deal with offset==0 erroneous edge case by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3937](https://togithub.com/facebook/zstd/pull/3937) - add tests inspired from [#​2927](https://togithub.com/facebook/zstd/issues/2927) by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3948](https://togithub.com/facebook/zstd/pull/3948) - cmake refactor: move HP-UX specific logic into its own function by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3946](https://togithub.com/facebook/zstd/pull/3946) - Fix [#​3719](https://togithub.com/facebook/zstd/issues/3719) : mixing -c, -o and --rm by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3942](https://togithub.com/facebook/zstd/pull/3942) - minor: fix incorrect debug level by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3936](https://togithub.com/facebook/zstd/pull/3936) - add RISC-V emulation tests to Github CI by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3934](https://togithub.com/facebook/zstd/pull/3934) - prevent XXH64 from being autovectorized by XXH512 by default by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3933](https://togithub.com/facebook/zstd/pull/3933) - Stop Hardcoding the POSIX Version on BSDs by [@​felixhandte](https://togithub.com/felixhandte) in [https://github.com/facebook/zstd/pull/3952](https://togithub.com/facebook/zstd/pull/3952) - Convert the CircleCI workflow to a GitHub Actions workflow by [@​jk0](https://togithub.com/jk0) in [https://github.com/facebook/zstd/pull/3901](https://togithub.com/facebook/zstd/pull/3901) - Add common compressed file types to --exclude-compressed by [@​daniellerozenblit](https://togithub.com/daniellerozenblit) in [https://github.com/facebook/zstd/pull/3951](https://togithub.com/facebook/zstd/pull/3951) - Export ZSTD_LEGACY_SUPPORT in tests/Makefile by [@​embg](https://togithub.com/embg) in [https://github.com/facebook/zstd/pull/3955](https://togithub.com/facebook/zstd/pull/3955) - Exercise ZSTD_findDecompressedSize() in the simple decompression fuzzer by [@​embg](https://togithub.com/embg) in [https://github.com/facebook/zstd/pull/3959](https://togithub.com/facebook/zstd/pull/3959) - Update `ZSTD_RowFindBestMatch` comment by [@​yoniko](https://togithub.com/yoniko) in [https://github.com/facebook/zstd/pull/3947](https://togithub.com/facebook/zstd/pull/3947) - Add the zeroSeq sample by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3954](https://togithub.com/facebook/zstd/pull/3954) - \[cpu] Backport fix for rbx clobbering on Windows with Clang by [@​terrelln](https://togithub.com/terrelln) in [https://github.com/facebook/zstd/pull/3957](https://togithub.com/facebook/zstd/pull/3957) - Do not truncate file name in verbose mode by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3956](https://togithub.com/facebook/zstd/pull/3956) - updated documentation by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3958](https://togithub.com/facebook/zstd/pull/3958) - \[asm]\[aarch64] Mark that BTI and PAC are supported by [@​terrelln](https://togithub.com/terrelln) in [https://github.com/facebook/zstd/pull/3961](https://togithub.com/facebook/zstd/pull/3961) - Use `utimensat()` on FreeBSD by [@​felixhandte](https://togithub.com/felixhandte) in [https://github.com/facebook/zstd/pull/3960](https://togithub.com/facebook/zstd/pull/3960) - reduce the amount of #include in cover.h by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3962](https://togithub.com/facebook/zstd/pull/3962) - Remove Erroneous Exclusion of Hidden Files and Folders in `--output-dir-mirror` by [@​felixhandte](https://togithub.com/felixhandte) in [https://github.com/facebook/zstd/pull/3963](https://togithub.com/facebook/zstd/pull/3963) - Promote `ZSTD_c_targetCBlockSize` Parameter to Stable API by [@​felixhandte](https://togithub.com/felixhandte) in [https://github.com/facebook/zstd/pull/3964](https://togithub.com/facebook/zstd/pull/3964) - \[cmake] Always create libzstd target by [@​terrelln](https://togithub.com/terrelln) in [https://github.com/facebook/zstd/pull/3965](https://togithub.com/facebook/zstd/pull/3965) - Remove incorrect docs regarding ZSTD_findFrameCompressedSize() by [@​embg](https://togithub.com/embg) in [https://github.com/facebook/zstd/pull/3967](https://togithub.com/facebook/zstd/pull/3967) - add line number to debug traces by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3966](https://togithub.com/facebook/zstd/pull/3966) - bump version number by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3969](https://togithub.com/facebook/zstd/pull/3969) - Export zstd's public headers via BUILD_INTERFACE by [@​terrelln](https://togithub.com/terrelln) in [https://github.com/facebook/zstd/pull/3968](https://togithub.com/facebook/zstd/pull/3968) - Fix bug with streaming decompression of magicless format by [@​embg](https://togithub.com/embg) in [https://github.com/facebook/zstd/pull/3971](https://togithub.com/facebook/zstd/pull/3971) - pzstd: use c++14 without conditions by [@​kanavin](https://togithub.com/kanavin) in [https://github.com/facebook/zstd/pull/3682](https://togithub.com/facebook/zstd/pull/3682) - Fix bugs in simple decompression fuzzer by [@​yoniko](https://togithub.com/yoniko) in [https://github.com/facebook/zstd/pull/3978](https://togithub.com/facebook/zstd/pull/3978) - Fuzzing and bugfixes for magicless-format decoding by [@​embg](https://togithub.com/embg) in [https://github.com/facebook/zstd/pull/3976](https://togithub.com/facebook/zstd/pull/3976) - Fix & fuzz ZSTD_generateSequences by [@​terrelln](https://togithub.com/terrelln) in [https://github.com/facebook/zstd/pull/3981](https://togithub.com/facebook/zstd/pull/3981) - Fail on errors when building fuzzers by [@​yoniko](https://togithub.com/yoniko) in [https://github.com/facebook/zstd/pull/3979](https://togithub.com/facebook/zstd/pull/3979) - \[cmake] Emit warnings for contradictory build settings by [@​terrelln](https://togithub.com/terrelln) in [https://github.com/facebook/zstd/pull/3975](https://togithub.com/facebook/zstd/pull/3975) - Document the process for adding a new fuzzer by [@​embg](https://togithub.com/embg) in [https://github.com/facebook/zstd/pull/3982](https://togithub.com/facebook/zstd/pull/3982) - Fix -Werror=pointer-arith in fuzzers by [@​embg](https://togithub.com/embg) in [https://github.com/facebook/zstd/pull/3983](https://togithub.com/facebook/zstd/pull/3983) - Doc update by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3977](https://togithub.com/facebook/zstd/pull/3977) - v1.5.6 by [@​Cyan4973](https://togithub.com/Cyan4973) in [https://github.com/facebook/zstd/pull/3984](https://togithub.com/facebook/zstd/pull/3984) #### New Contributors - [@​Kim-SSi](https://togithub.com/Kim-SSi) made their first contribution in [https://github.com/facebook/zstd/pull/3600](https://togithub.com/facebook/zstd/pull/3600) - [@​mredig](https://togithub.com/mredig) made their first contribution in [https://github.com/facebook/zstd/pull/3614](https://togithub.com/facebook/zstd/pull/3614) - [@​dunhor](https://togithub.com/dunhor) made their first contribution in [https://github.com/facebook/zstd/pull/3636](https://togithub.com/facebook/zstd/pull/3636) - [@​sighingnow](https://togithub.com/sighingnow) made their first contribution in [https://github.com/facebook/zstd/pull/3657](https://togithub.com/facebook/zstd/pull/3657) - [@​nidhijaju](https://togithub.com/nidhijaju) made their first contribution in [https://github.com/facebook/zstd/pull/3688](https://togithub.com/facebook/zstd/pull/3688) - [@​gjasny](https://togithub.com/gjasny) made their first contribution in [https://github.com/facebook/zstd/pull/3665](https://togithub.com/facebook/zstd/pull/3665) - [@​Coder-256](https://togithub.com/Coder-256) made their first contribution in [https://github.com/facebook/zstd/pull/3700](https://togithub.com/facebook/zstd/pull/3700) - [@​LocutusOfBorg](https://togithub.com/LocutusOfBorg) made their first contribution in [https://github.com/facebook/zstd/pull/3695](https://togithub.com/facebook/zstd/pull/3695) - [@​void0red](https://togithub.com/void0red) made their first contribution in [https://github.com/facebook/zstd/pull/3704](https://togithub.com/facebook/zstd/pull/3704) - [@​nikohoffren](https://togithub.com/nikohoffren) made their first contribution in [https://github.com/facebook/zstd/pull/3701](https://togithub.com/facebook/zstd/pull/3701) - [@​alexsifivetw](https://togithub.com/alexsifivetw) made their first contribution in [https://github.com/facebook/zstd/pull/3712](https://togithub.com/facebook/zstd/pull/3712) - [@​0o001](https://togithub.com/0o001) made their first contribution in [https://github.com/facebook/zstd/pull/3728](https://togithub.com/facebook/zstd/pull/3728) - [@​QBos07](https://togithub.com/QBos07) made their first contribution in [https://github.com/facebook/zstd/pull/3720](https://togithub.com/facebook/zstd/pull/3720) - [@​JohanMabille](https://togithub.com/JohanMabille) made their first contribution in [https://github.com/facebook/zstd/pull/3739](https://togithub.com/facebook/zstd/pull/3739) - [@​klausholstjacobsen](https://togithub.com/klausholstjacobsen) made their first contribution in [https://github.com/facebook/zstd/pull/3745](https://togithub.com/facebook/zstd/pull/3745) - [@​Saverio976](https://togithub.com/Saverio976) made their first contribution in [https://github.com/facebook/zstd/pull/3795](https://togithub.com/facebook/zstd/pull/3795) - [@​elasota](https://togithub.com/elasota) made their first contribution in [https://github.com/facebook/zstd/pull/3806](https://togithub.com/facebook/zstd/pull/3806) - [@​jondo2010](https://togithub.com/jondo2010) made their first contribution in [https://github.com/facebook/zstd/pull/3812](https://togithub.com/facebook/zstd/pull/3812) - [@​gruenich](https://togithub.com/gruenich) made their first contribution in [https://github.com/facebook/zstd/pull/3807](https://togithub.com/facebook/zstd/pull/3807) - [@​michoecho](https://togithub.com/michoecho) made their first contribution in [https://github.com/facebook/zstd/pull/3847](https://togithub.com/facebook/zstd/pull/3847) - [@​KapJI](https://togithub.com/KapJI) made their first contribution in [https://github.com/facebook/zstd/pull/3850](https://togithub.com/facebook/zstd/pull/3850) - [@​jcelerier](https://togithub.com/jcelerier) made their first contribution in [https://github.com/facebook/zstd/pull/3760](https://togithub.com/facebook/zstd/pull/3760) - [@​sandreenko](https://togithub.com/sandreenko) made their first contribution in [https://github.com/facebook/zstd/pull/3899](https://togithub.com/facebook/zstd/pull/3899) - [@​teo-tsirpanis](https://togithub.com/teo-tsirpanis) made their first contribution in [https://github.com/facebook/zstd/pull/3811](https://togithub.com/facebook/zstd/pull/3811) - [@​aimuz](https://togithub.com/aimuz) made their first contribution in [https://github.com/facebook/zstd/pull/3840](https://togithub.com/facebook/zstd/pull/3840) - [@​acceptacross](https://togithub.com/acceptacross) made their first contribution in [https://github.com/facebook/zstd/pull/3949](https://togithub.com/facebook/zstd/pull/3949) - [@​jk0](https://togithub.com/jk0) made their first contribution in [https://github.com/facebook/zstd/pull/3901](https://togithub.com/facebook/zstd/pull/3901) **Full Changelog**: facebook/zstd@v1.5.5...v1.5.6 </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://developer.mend.io/github/secretflow/spu). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy4yNjkuMiIsInVwZGF0ZWRJblZlciI6IjM3LjI2OS4yIiwidGFyZ2V0QnJhbmNoIjoibWFpbiJ9--> Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
](https://renovatebot.com){kind=link}
|
Hi @Cyan4973, excellent work! Could you please clarify which "High compression mode" levels benefit from this work? Only I tried to figure this out from the changelog/code diff, but either I am missing it or it is just unclear... |
|
It think it's active for strategies |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is an answer to issues similar to #3793 and #3014,
which report a poor compression ratio at high compression level
on synthetic data, typically arrays of 32-bit integers with very little consecutive variation (like an increasing series).
While such synthetic scenario tends to be "too regular", and therefore not representative of less regular "real" data,
it's nonetheless frequently used as a test scenario, because it is easy to produce.
The High compression mode has indeed a weakness in such a scenario,
and this patch tries to address it.
A first patch was able to improve these synthetic scenarios, increasing compression ratio by a significant amount. But unfortunately, it would then lose some compression ratio for other test data.
This patch is a variant, which manages to improve compression ratio in the case of 32-bit integer arrays, without losing anything in other cases.
This is illustrated in below test cases :
devPR.debug_str_offsetsAs can be seen, compression benefits are very sensible on the synthetic examples. More realistic samples (such as standard compression corpuses) get lower yet noticeable gains, notably
.debug_str_offsets,kennedy.xls,geoandx-ray. Several other files get smaller gains (+0.1% range), but more importantly, no file in any compression corpus is negatively impacted by this patch (beyond a few bytes here and there just as a consequence of random luck).Overall, the resulting outcome of this patch on
silesiaat level 19 is > 100 KB savings.As a side note, I was hoping for an even better outcome because the gains from an earlier patch for 32-bit integer arrays were larger. But this previous patch was also negatively impacting compression ratio of other (non 32-bit integers) files. Therefore I believe this positive-only patch is a more acceptable trade-off for merge.
I might try later to reproduce the stronger results of the first patch (on 32-bit integer arrays), with the hope to find a solution which doesn't negatively impact other data. But this may require a complete overhaul of the optimal parser, and in the meantime, we can at least use this patch.