LZ5 v2.0 beta

[inikep]

LZ5 v2.0 beta can be downloaded from https://github.com/inikep/lz5/releases

Changes:
- this version is optimized for decompression speed (instead of ratio for previous versions)
- levels from 1 to 9 use LZ4-type codewords and give about 10% better decompression speed than LZ4
- levels from 10 to 18 use new codewords with decompression speed about 2 times faster than LZ5 v1.5

The results below are performed using the same laptop with the same compiler (gcc 6.2.0 MinGW-w64):

LZ5 v2.0 beta compiled from latest "lz5_v2.0" branch:
 1#silesia_tar       : 211947520 -> 103402971 (2.050), 346.5 MB/s ,2577.0 MB/s
 2#silesia_tar       : 211947520 ->  93861621 (2.258), 251.0 MB/s ,2412.5 MB/s
 3#silesia_tar       : 211947520 ->  89443564 (2.370), 118.0 MB/s ,2430.6 MB/s
 4#silesia_tar       : 211947520 ->  86646242 (2.446),  72.0 MB/s ,2480.6 MB/s
 5#silesia_tar       : 211947520 ->  83975809 (2.524),  58.6 MB/s ,2525.3 MB/s
 6#silesia_tar       : 211947520 ->  81223047 (2.609),  56.2 MB/s ,2521.2 MB/s
 7#silesia_tar       : 211947520 ->  79381197 (2.670),  40.7 MB/s ,2402.4 MB/s
 8#silesia_tar       : 211947520 ->  78021788 (2.717),  19.5 MB/s ,2448.3 MB/s
 9#silesia_tar       : 211947520 ->  77588075 (2.732),   2.8 MB/s ,2460.8 MB/s
10#silesia_tar       : 211947520 ->  89799372 (2.360), 171.1 MB/s ,1749.6 MB/s
11#silesia_tar       : 211947520 ->  85225312 (2.487), 107.2 MB/s ,1674.6 MB/s
12#silesia_tar       : 211947520 ->  82302947 (2.575),  33.5 MB/s ,1777.8 MB/s
13#silesia_tar       : 211947520 ->  79328026 (2.672),  21.8 MB/s ,1748.3 MB/s
14#silesia_tar       : 211947520 ->  75818444 (2.795),  13.1 MB/s ,1699.0 MB/s
15#silesia_tar       : 211947520 ->  74584415 (2.842),   8.3 MB/s ,1684.3 MB/s
16#silesia_tar       : 211947520 ->  72979690 (2.904),   4.5 MB/s ,1714.6 MB/s
17#silesia_tar       : 211947520 ->  70500121 (3.006),   3.0 MB/s ,1620.6 MB/s
18#silesia_tar       : 211947520 ->  68539649 (3.092),   1.5 MB/s ,1502.5 MB/s

LZ4 v1.7.2 compiled from latest "dev" branch:
 1#esty\silesia_tar : 211947520 -> 100923164 (47.62%),  437.9 MB/s , 2265.7 MB/s
 2#esty\silesia_tar : 211947520 -> 100923164 (47.62%),  435.2 MB/s , 2254.8 MB/s
 3#esty\silesia_tar : 211947520 ->  82428029 (38.89%),   66.1 MB/s , 2150.8 MB/s
 4#esty\silesia_tar : 211947520 ->  80589674 (38.02%),   54.7 MB/s , 2201.5 MB/s
 5#esty\silesia_tar : 211947520 ->  79427759 (37.48%),   45.1 MB/s , 2209.9 MB/s
 6#esty\silesia_tar : 211947520 ->  78747219 (37.15%),   37.3 MB/s , 2235.3 MB/s
 7#esty\silesia_tar : 211947520 ->  78354189 (36.97%),   31.0 MB/s , 2248.2 MB/s
 8#esty\silesia_tar : 211947520 ->  78140269 (36.87%),   26.3 MB/s , 2239.6 MB/s
 9#esty\silesia_tar : 211947520 ->  78036683 (36.82%),   23.8 MB/s , 2243.9 MB/s

LZ5 v1.5.1 compiled from latest "lz5_v1.5" branch:
 1#silesia_tar       : 211947520 -> 113538427 (1.867), 428.7 MB/s ,1304.2 MB/s
 2#silesia_tar       : 211947520 -> 104082573 (2.036), 346.6 MB/s ,1251.9 MB/s
 3#silesia_tar       : 211947520 ->  95575943 (2.218), 261.6 MB/s ,1129.6 MB/s
 4#silesia_tar       : 211947520 ->  86503541 (2.450), 149.2 MB/s , 957.5 MB/s
 5#silesia_tar       : 211947520 ->  84460413 (2.509), 115.0 MB/s , 884.6 MB/s
 6#silesia_tar       : 211947520 ->  80575745 (2.630),  42.3 MB/s , 972.8 MB/s
 7#silesia_tar       : 211947520 ->  78854960 (2.688),  37.2 MB/s , 940.9 MB/s
 8#silesia_tar       : 211947520 ->  77996518 (2.717),  27.3 MB/s , 915.4 MB/s
 9#silesia_tar       : 211947520 ->  74228639 (2.855),  20.3 MB/s , 917.8 MB/s
10#silesia_tar       : 211947520 ->  72267610 (2.933),  13.4 MB/s , 916.6 MB/s
11#silesia_tar       : 211947520 ->  71056249 (2.983),   9.7 MB/s , 824.1 MB/s
12#silesia_tar       : 211947520 ->  69485691 (3.050),   7.6 MB/s , 837.8 MB/s
13#silesia_tar       : 211947520 ->  67937025 (3.120),   4.5 MB/s , 864.9 MB/s
14#silesia_tar       : 211947520 ->  65805781 (3.221),   2.9 MB/s , 851.7 MB/s
15#silesia_tar       : 211947520 ->  65555479 (3.233),   1.6 MB/s , 868.3 MB/s

[inikep]

Do you have any comments on v2.0 beta?

[xcrh]

I've stumbled on it recently but only had time to do several quick runs only on x86-64 laptop (i.e. no fun with ARM things and so on) and only got overall impression. On x86_64 its speed looks impressive, but it seems achievable ratio has suffered a bit. Also, it comes with quite unusual bitstream format. Am I correct that decompression basically looks like this: get size of stream - go to current + size, get size of next stream, etc, etc for all 5 streams? Does such design leads to better speeds? (and why? better cpu cache hits?)

I've also spotted some few issues/inconsistences. Do you prefer 'em to get listed here or filed on github as "issues"?

Also I have question: do you have some plans to add lz5 v2 to lzbench? I think I've mostly understood what it takes to add new codec to lzbench (and since LZ5v2 differs from 1.5 considerably IMHO it better to be new codec?), so I guess I could try something like that, but I would dislike if it interferes with your own plans and I'm not exactly sure I could do it properly on first try, I could miss something for sure :P.

[inikep]

Also, it comes with quite unusual bitstream format. Am I correct that decompression basically looks like this: get size of stream - go to current + size, get size of next stream, etc, etc for all 5 streams? Does such design leads to better speeds? (and why? better cpu cache hits?)

The speed is similar to 1 stream but it helps for Huffman encoding. You can try it using "make lz5huf" and then "lz5huf -l3".

Do you prefer 'em to get listed here or filed on github as "issues"?

Github should be better.

Also I have question: do you have some plans to add lz5 v2 to lzbench?

Sure, I will add lz5v2 to lzbench. I was thinking about replacing v1.5 with v2.0. Do you think that I should keep also v1.5?

[xcrh]

The speed is similar to 1 stream but it helps for Huffman encoding. You can try it using "make lz5huf" and then "lz5huf -l3".

I've seen some huff-related things, but straightforward attempts to use it lead me to few things which are probably bugs

Github should be better.

Ok, then I'll go here.

Sure, I will add lz5v2 to lzbench. I was thinking about replacing v1.5 with v2.0. Do you think that I should keep also v1.5?

After few quick runs I've got idea LZ5v2 offers somewhat different tradeoffs, being a bit less dense but considerably faster (at least on my x86_64 laptop and few desktops; I haven't ran tests on ARM things yet, etc). So I've got idea it could be interesting to run 'em side by side.