lzbench - in-memory benchmark of open-source LZ77/LZSS/LZMA compressors

[inikep]

lzbench v1.3 has been released https://github.com/inikep/lzbench/releases

Changes:
- added slz 1.0.0 : slz_zlib, slz_gzip, slz_deflate (thanks to Willy Tarreau)
- added glza 0.7.1 (thanks to Kennon Conrad)
- zstd updated to v0.8.0 (thanks to Chip Turner)
- brotli updated to v0.5.2
- gipfeli updated to 2016-07-13
- lzfse and lzvn updated to 2016-08-16
- lz5 updated to v1.5
- lzbench compiles on MacOS (with exception of lzham)
- lzbench compiles with clang (tested with 3.4, 3.5, 3.6 and 3.8 )
- added support for aggregated parameters: "-t0 -u0 -i3 -j5" is equal to "-t0u0i3j5" (only "-e" can not be aggregated)
- added new options:

-r = disable real-time process priority
-v = disable progress information
-z = show (de)compression times instead of speed

[inikep]

lzbench v1.4 has been released at https://github.com/inikep/lzbench/releases

Changes:
- zstd updated to v1.0.0
- added libdeflate 2016-08-29 (thanks to Jørgen Ibsen)
- added the "-m" (memory_limit) option; large files are processed in (memory_limit/4) blocks
- cleaner CSV output and added the "original size" field

[inikep]

lzbench v1.5 has been released at https://github.com/inikep/lzbench/releases

Changes:
- added the "-r" option : operate recursively on directories
- added the "-j" option : join files in memory but compress them independently (for many small files)
- lz5 updated to v2.0 RC2
- lz4 updated to v1.7.3
- zstd updated to v1.1.1
- libdeflate updated to v0.6
- csc updated to 2016-10-13
- improved PowerPC support
- fixed "-m" option (thanks to Julian Kunkel)

[Jyrki Alakuijala]

lzbench v1.5 has been released

Would you consider adding the large window brotli?

I'm thinking of basing the brotli framing format on a brotli extension like it, and it would be nice to learn more about its performance with a benchmarking tool like lzbench.

For silensia.tar, large window brotli compresses down to 50164028 bytes (23.67 %) with 30 bit window. A 24 bit window (normal brotli) gives 50397564 bytes (23.78 %). For other more homogeneous benchmarking files, such as enwik9, the savings from large window can be 10 %.

Btw, you are comparing a 22 bit windowed brotli against 23 bit windowed zstd here: https://github.com/inikep/lzbench/ -- This can be a compression density difference of 4-5 %, but probably only 1 % for silesia.tar. It would be a more interesting comparison if all compression algorithms are run with the same window size.

[inikep]

Would you consider adding the large window brotli?

I don't know details of large window brotli but I'm afraid that there will be a collision of symbols/function names. Is there any simple solution (e.g. #define) for that?

you are comparing a 22 bit windowed brotli against 23 bit windowed zstd

No, I'm comparing default compression levels (e.g. lzham has 26-bit window).

It would be a more interesting comparison if all compression algorithms are run with the same window size.

It's already done. There are aliases with 22 and 24-bit window: brotli22, brotli24, lzham22, lzham24, zstd22, zstd24 so you can try e.g.

lzbench -ebrotli22/zstd22 silesia.tar

[inikep]

The following results are obtained with "lzbench -t16,16 -eall" using 1 core of Intel Core i5-4300U, Windows 10 64-bit (MinGW-w64 compilation under gcc 6.2.0) with silesia.tar.
Full results are available here: https://github.com/inikep/lzbench/bl...ch15_sorted.md

The pareto frontier for decompression speed and ratio:

| Compressor name         | Compression| Decompress.| Compr. size | Ratio |
| ---------------         | -----------| -----------| ----------- | ----- |
| lzlib 1.7 -9            |  1.21 MB/s |    45 MB/s |    48296889 | 22.79 |
| xz 5.2.2 -9             |  1.80 MB/s |    58 MB/s |    48745306 | 23.00 |
| xz 5.2.2 -6             |  1.97 MB/s |    58 MB/s |    49195929 | 23.21 |
| lzma 9.38 -5            |  2.08 MB/s |    65 MB/s |    49720569 | 23.46 |
| brotli 0.5.2 -11        |  0.39 MB/s |   266 MB/s |    51138054 | 24.13 |
| zstd 1.1.1 -22          |  1.44 MB/s |   505 MB/s |    52731930 | 24.88 | 
| zstd 1.1.1 -18          |  2.87 MB/s |   583 MB/s |    55294241 | 26.09 |
| zstd 1.1.1 -15          |  4.97 MB/s |   639 MB/s |    58007773 | 27.37 |
| lz5 2.0 RC2 -49         |  1.27 MB/s |  1064 MB/s |    60679215 | 28.63 |
| lz5 2.0 RC2 -45         |  6.25 MB/s |  1073 MB/s |    65413061 | 30.86 |
| lz5 2.0 RC2 -29         |  1.37 MB/s |  1634 MB/s |    68694227 | 32.41 |
| lz5 2.0 RC2 -25         |  6.63 MB/s |  1734 MB/s |    74503695 | 35.15 |
| lzsse8 2016-05-14 -16   |  6.05 MB/s |  2858 MB/s |    75464339 | 35.61 |
| lzsse8 2016-05-14 -12   |  6.00 MB/s |  2859 MB/s |    75464339 | 35.61 |
| blosclz 2015-11-10 -3   |   487 MB/s |  5047 MB/s |   204507781 | 96.49 |
| blosclz 2015-11-10 -1   |   892 MB/s |  5679 MB/s |   211768481 | 99.92 |
| memcpy                  |  7332 MB/s |  8719 MB/s |   211947520 |100.00 |

The pareto frontier for compression speed and ratio:

| Compressor name         | Compression| Decompress.| Compr. size | Ratio |
| ---------------         | -----------| -----------| ----------- | ----- |
| lzlib 1.7 -9            |  1.21 MB/s |    45 MB/s |    48296889 | 22.79 |
| xz 5.2.2 -9             |  1.80 MB/s |    58 MB/s |    48745306 | 23.00 |
| xz 5.2.2 -6             |  1.97 MB/s |    58 MB/s |    49195929 | 23.21 |
| lzma 9.38 -5            |  2.08 MB/s |    65 MB/s |    49720569 | 23.46 |
| csc 2016-10-13 -5       |  2.49 MB/s |    50 MB/s |    49801577 | 23.50 |
| csc 2016-10-13 -3       |  5.98 MB/s |    46 MB/s |    53477914 | 25.23 |
| csc 2016-10-13 -1       |    14 MB/s |    48 MB/s |    56201092 | 26.52 |
| lzma 9.38 -2            |    15 MB/s |    55 MB/s |    58867911 | 27.77 |
| zstd 1.1.1 -11          |    16 MB/s |   613 MB/s |    59523167 | 28.08 |
| brotli 0.5.2 -5         |    24 MB/s |   312 MB/s |    60801716 | 28.69 |
| zling 2016-01-10 -4     |    26 MB/s |   135 MB/s |    60997465 | 28.78 |
| zstd 1.1.1 -8           |    31 MB/s |   619 MB/s |    61026497 | 28.79 |
| zling 2016-01-10 -2     |    32 MB/s |   135 MB/s |    61917662 | 29.21 |
| zling 2016-01-10 -1     |    36 MB/s |   135 MB/s |    62438620 | 29.46 |
| zling 2016-01-10 -0     |    40 MB/s |   132 MB/s |    63407921 | 29.92 |
| zstd 1.1.1 -5           |    88 MB/s |   565 MB/s |    65002208 | 30.67 |
| brotli 0.5.2 -2         |    96 MB/s |   283 MB/s |    68066621 | 32.11 |
| zstd 1.1.1 -2           |   181 MB/s |   600 MB/s |    70168955 | 33.11 |
| zstd 1.1.1 -1           |   235 MB/s |   645 MB/s |    73659468 | 34.75 |
| lz5 2.0 RC2 -30         |   246 MB/s |   909 MB/s |    85727429 | 40.45 |
| density 0.12.5 beta -3  |   254 MB/s |   234 MB/s |    87622980 | 41.34 |
| pithy 2011-12-24 -9     |   257 MB/s |  1269 MB/s |    90360813 | 42.63 |
| pithy 2011-12-24 -6     |   297 MB/s |  1285 MB/s |    92090898 | 43.45 |
| quicklz 1.5.0 -1        |   349 MB/s |   439 MB/s |    94720562 | 44.69 |
| pithy 2011-12-24 -3     |   354 MB/s |  1235 MB/s |    97255186 | 45.89 |
| lzo1x 2.09 -1           |   401 MB/s |   548 MB/s |   100572537 | 47.45 |
| lz4 1.7.3               |   440 MB/s |  2318 MB/s |   100880800 | 47.60 |
| density 0.12.5 beta -2  |   479 MB/s |   660 MB/s |   101706226 | 47.99 |
| lz4fast 1.7.3 -3        |   516 MB/s |  2317 MB/s |   107066190 | 50.52 |
| lz4fast 1.7.3 -17       |   781 MB/s |  2696 MB/s |   131732802 | 62.15 |
| density 0.12.5 beta -1  |   806 MB/s |  1028 MB/s |   133085162 | 62.79 |

[m^3]

I didn't realize that the field of moderately fast to fast compressors (judging by compression speed) was this competitive.

[Jyrki Alakuijala]

The pareto frontier for ...

A pareto-analysis would be more interesting if all compressors are measured with the same window size -- or alternatively would run the compressors with a large variety of allowed window sizes. Now both brotli and zstd are taking a huge hit against the others when they are run at 22 and 23 bits against possibly 28 bit? window of lzma.

(A smaller window size means better n-way multi-processing and less resource use during streaming, but neither of these characteristics are measured in this benchmark.)

[inikep]

A pareto-analysis would be more interesting if all compressors are measured with the same window size.

I understand that the same window size (or even better: the same memory usage) might be useful for multitasking or servers with the specified limit of memory but there are also many other applications of data compression. I also have a few comments:
1. Does it mean that you prefer a 16-bit window size to match LZ4/lzsse/snappy?
2. Zstd has a different window size almost for each compression level. It even changes parameters depending on file size. Zstd -22 currently uses a 27-bit window. Forcing LZMA/XZ/Zstd to 22-bit or 24-bit could influence other parameters and destroy ratio/speed tradeoff designed for this algorithm.
3. I'm open to accept this kind of patch for lzbench. I have no time and belief to do it by myself.

[Jyrki Alakuijala]

I understand that the same window size (or even better: the same memory usage) might be useful for multitasking or servers with the specified limit of memory but there are also many other applications of data compression. I also have a few comments:
1. Does it mean that you prefer a 16-bit window size to match LZ4/lzsse/snappy?

I think for every algorithm the only thing that matters is that what brings it to the pareto-frontier. If lower or higher window sizes bring an algorithm to the pareto-frontier, then tuning the window size can be important.

Other thoughts on this:

I don't think < 18 bits matters much in resource use. There is usually a 256+ kB dedicated cache per core nowadays, and most software that wants to decompress can afford a 256 kB buffer for it.

Of course, it is technically impossible to use higher windows with some of these formats, like snappy and zlib are only up to 15 bits.

2. Zstd has a different window size almost for each compression level. It even changes parameters depending on file size. Zstd -22 currently uses a 27-bit window. Forcing LZMA/XZ/Zstd to 22-bit or 24-bit could influence other parameters and destroy ratio/speed tradeoff designed for this algorithm.

I don't think there is anything magical in LZMA etc that tunes to the window size. Here, a simple comparison on the window size impact on silesia.tar with lzma:9 and brotli:11

lzma:9 48933295 49675940 51116450 (window size 24, 22 and 20)
4.27 % between 24 bit and 20 bit window
2.82 % between 22 bit and 20 bit window

brotli:11 50397564 51139327 52632475 (window size 24, 22 and 20)
4.25 % between 24 bit and 20 bit window
2.84 % between 22 bit and 20 bit window

I believe it is safe to assume that LZ77-based approaches benefit from a larger window size in a similar manner, and that it is harmful to compare them using different window sizes in different algorithms.

[inikep]

I think for every algorithm the only thing that matters is that what brings it to the pareto-frontier.

The pareto frontier shows only selected properties of the algorithm. In my results one can see quite different algorithms in the pareto frontier for compression speed/ratio and decompression speed/ratio. Moreover it depends on used computer, input files, compiler, and so on.
The bigger issue with my pareto results is that "lzbench -eall" selects every 3rd ot 4th compression level for most of compressors because there are so many compressors and levels (look at lzo or lz5 which now has 40 levels). So it's also the pareto frontier limited to compressors available in lzbench with limited amount of levels. But everybody is invited to make their own tests.

that it is harmful to compare them using different window sizes in different algorithms.

I think it's valid when you're taking only ratio into account. A different window size highly influences on compression speed and may destroy ratio/speed tradeoff designed by author for this algorithm. Look for example at https://github.com/facebook/zstd/blo....c#L3130-L3152

[inikep]

lzbench v1.6 has been released at https://github.com/inikep/lzbench/releases

Changes:
- zstd updated to v1.1.3
- lz4 updated to v1.7.5
- lz5 updated to v2.0
- improved: touch pages in buffer allocations prior to running first algorithm (thanks to travisdowns)
- fixed: check if input name is a directory
- fixed: median time (the "-p3" option, thanks to Alexey Tourbin)

[Samantha]

Hi @inikep, wanted to know if there were updates for lzbench 1.6, is whether there was the possibility of adding other compressors not lz, example (FB9, FP8, nanozip, paq8px, zcm and zpaq).
I fixed through the GUI, the reading limit for large files by calculating the average of the divided parts of the file, for each test bench.
At the end I used lzbench 1.6 for a wide range of compressors and stability.
If you want to try my utility let me know, I'll send in PM.



Thanks again for your work.

[inikep]

possibility of adding other compressors not lz, example (FB9, FP8, nanozip, paq8px, zcm and zpaq)

It's not possible to add nanozip and zcm because these are closed-source compressors.
I think that I will not add non-LZ compressors by myself but I will accept them as pull requests.
More information at https://github.com/inikep/lzbench/issues/21

[Samantha]

OK, I hope in the future there are new updates, like the ability to add an option with only compression speed for each scanned file, instead of both options, in this mode, the scan will be even faster for files of large size.

[RamiroCruzo]

Greetings,
I was reading lzbench's readme on GitHub where you've said that you can't add more compressors because of unavailability of source code. You can just create an option for piped compressor where a user can define compression and decompression commands, of course, it'll only work for Softwares with stdio, still, it'll add a lot of more compressors in the benchmark.

[Samantha]

Hi @Ramiro, Well yes, it certainly is an idea, but I'd like to have support from @Inikep, so it can improve and expand lzb1.6 version with more options, or in any way get the possibility to test compressors not lz. I could get a perfect GUI able to test all the compressors, in an easy, fast and intuitive, at your disposal for all users, here or on FF.

[inikep]

OK, I hope in the future there are new updates, like the ability to add an option with only compression speed for each scanned file, instead of both options, in this mode, the scan will be even faster for files of large size.

I've added "--compress-only" option in https://github.com/inikep/lzbench/co...32405df9d6b05d.
If you want to have a single compression call for each compressor you can use "--compress-only -t0".

You can just create an option for piped compressor where a user can define compression and decompression commands, of course, it'll only work for Softwares with stdio, still, it'll add a lot of more compressors in the benchmark.

Testing executables has many disadvantages:
- instead of testing memory-to-memory compression speed we are also taking input/output into consideration
- executable usually will be compiled with a different compiler and can be compiled with different optimizations
- do we want to keep executables within GitHub repo? (are we allowed?)

[Samantha]

Hi inikep you have read the P.M. ?

[Samantha]

Hi Inikep thank you bro, a small clarification if it was possible, with the "--compress -t0-only" option I get the message "Error" under name decompression, you could write "Disabled" instead of "Error", but does the same I will not disturb you any more time...

[inikep]

lzbench v1.7 has been released at https://github.com/inikep/lzbench/releases

Changes:
- added lizard v1.0 (formerly lz5)
- libdeflate updated to v0.7
- lzfse and lzvn updated to 2017-03-08
- lzlib updated to v1.8
- snappy updated to v1.1.4
- xz updated to v5.2.3
- zlib updated to v1.2.11
- zstd updated to v1.1.4

[Jyrki Alakuijala]

lzbench v1.7 has been released at https://github.com/inikep/lzbench/releases

Changes:
- added lizard v1.0 (formerly lz5)
- libdeflate updated to v0.7
- lzfse and lzvn updated to 2017-03-08
- lzlib updated to v1.8
- snappy updated to v1.1.4
- xz updated to v5.2.3
- zlib updated to v1.2.11
- zstd updated to v1.1.4

There is a nice improvement in brotli compression at mid to high qualities and large data (1 MB+). Consider updating it, too?

[Samantha]

excellent Inikep, I expecting an update, sorry curiosity, but why change the name from LZ5 a lizard... for what reason?

[inikep]

why change the name from LZ5 a lizard... for what reason?

https://encode.su/threads/2721-Do-yo...uld-be-changed

There is a nice improvement in brotli compression at mid to high qualities and large data (1 MB+). Consider updating it, too?

I can do it but I was waiting for a new release at https://github.com/google/brotli/releases

[Samantha]

[QUOTE=inikep;52124]https://encode.su/threads/2721-Do-yo...uld-be-changed

Blheeee... long names do not like for compressors.....now I have to change part of the code for this nonsense...

[inikep]

lzbench v1.7.1 has been released at https://github.com/inikep/lzbench/releases
I also updated results with the pareto frontier for compression speed/ratio and decompression speed/ratio at https://github.com/inikep/lzbench/bl...h171_sorted.md

v1.7.1
- brotli updated to 2017-03-10
- glza updated to v0.8
- lzma updated to v16.04

v1.7
- added lizard v1.0 (formerly lz5)
- libdeflate updated to v0.7
- lzfse and lzvn updated to 2017-03-08
- lzlib updated to v1.8
- snappy updated to v1.1.4
- xz updated to v5.2.3
- zlib updated to v1.2.11
- zstd updated to v1.1.4

[Shelwien]

Why lzlib can use fb273 and lzma can't?

[inikep]

Why lzlib can use fb273 and lzma can't?

In lzbench I'm testing only predefined levels and lzma -9 is the highest level. I added results of lzma 16.04 -9.

[Shelwien]

I think its not enough, and its very misleading.
lzip is lzma port, so it can't compress that much better than lzma.
lzma defaults are clearly optimized for speed, and it never uses fb>64.
Compare these:
https://github.com/inikep/lzbench/bl...ssors.cpp#L568
https://github.com/inikep/lzbench/bl.../LzmaEnc.c#L82

I'd suggest explicitly setting fb273 at least for higher levels, in lzbench handler.
Also, LzmaEncProps_Normalize most likely has to be completely replaced - atm it only uses two threads at level 5+.
Higher lc could make sense in many cases (while it can't be changed in lzlib), etc.

Btw, I've also been wondering about defining parameter profiles for these "levels" -
is there some theory behind it? Like, exponential growth of some composite metric?

[inikep]

Why it's misleading? The executable of lzip -9 gives better compression than lzma -9.
This is the same issue as supporting a large window for Brotli. It can be solved with:
1. Modifying existing levels or adding new levels to compressors but it might be very confusing for users (IMHO not a good idea).
2. Adding a support for advanced compression options ("lzbench -elzma,9:fb=273") but it requires a lot of work.