Page 1 of 2 12 LastLast
Results 1 to 30 of 42

Thread: LZ5 v2.x - efficient compression with very fast decompression

  1. #1
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    309
    Thanks
    68
    Thanked 173 Times in 64 Posts

    Lightbulb LZ5 v2.x - efficient compression with very fast decompression

    LZ5 v2.0 Release Candidate can be downloaded from https://github.com/inikep/lz5/releases

    This version includes LZ5HUF which is LZ5 with additional Huffman compression of literals and tokens.

    Changes in v2.0:
    - this version is optimized for decompression speed (instead of ratio for previous versions)
    - levels from 1 to 9 use LZ4-type codewords and give about 10% better decompression speed than LZ4
    - levels from 10 to 18 use new codewords with decompression speed about 2 times faster than LZ5 v1.5

    The results of LZ5HUF (gcc 6.2.0 MinGW-w64, Intel Core i5-4300U):
    Code:
     1#silesia_tar       : 211947520 ->  85727429 (2.472), 261.9 MB/s , 908.3 MB/s
     2#silesia_tar       : 211947520 ->  81688522 (2.595), 203.0 MB/s ,1166.0 MB/s
     3#silesia_tar       : 211947520 ->  77653547 (2.729), 107.0 MB/s ,1160.1 MB/s
     4#silesia_tar       : 211947520 ->  75875936 (2.793),  66.5 MB/s ,1214.8 MB/s
     5#silesia_tar       : 211947520 ->  74460374 (2.846),  55.1 MB/s ,1310.4 MB/s
     6#silesia_tar       : 211947520 ->  73865643 (2.869),  52.9 MB/s ,1432.5 MB/s
     7#silesia_tar       : 211947520 ->  71841346 (2.950),  39.8 MB/s ,1457.4 MB/s
     8#silesia_tar       : 211947520 ->  70926099 (2.988),  19.6 MB/s ,1510.9 MB/s
     9#silesia_tar       : 211947520 ->  70933375 (2.988),   2.7 MB/s ,1564.2 MB/s
    10#silesia_tar       : 211947520 ->  76171036 (2.783), 135.8 MB/s , 941.4 MB/s
    11#silesia_tar       : 211947520 ->  73342615 (2.890),  91.8 MB/s , 975.2 MB/s
    12#silesia_tar       : 211947520 ->  71045063 (2.983),  30.5 MB/s ,1084.6 MB/s
    13#silesia_tar       : 211947520 ->  68888287 (3.077),  20.9 MB/s ,1098.8 MB/s
    14#silesia_tar       : 211947520 ->  66724919 (3.176),  12.8 MB/s ,1111.9 MB/s
    15#silesia_tar       : 211947520 ->  65418967 (3.240),   6.6 MB/s ,1062.6 MB/s
    16#silesia_tar       : 211947520 ->  64129093 (3.305),   3.8 MB/s ,1119.4 MB/s
    17#silesia_tar       : 211947520 ->  62114878 (3.412),   2.7 MB/s ,1087.4 MB/s
    18#silesia_tar       : 211947520 ->  60688211 (3.492),   1.2 MB/s ,1056.5 MB/s
    The results of LZ5 v2.0 RC (the same laptop and compiler):
    Code:
     1#silesia_tar       : 211947520 -> 103402971 (2.050), 354.6 MB/s ,2553.5 MB/s
     2#silesia_tar       : 211947520 ->  93861621 (2.258), 253.3 MB/s ,2390.7 MB/s
     3#silesia_tar       : 211947520 ->  89443564 (2.370), 115.0 MB/s ,2416.9 MB/s
     4#silesia_tar       : 211947520 ->  86646242 (2.446),  72.2 MB/s ,2454.6 MB/s
     5#silesia_tar       : 211947520 ->  83975809 (2.524),  59.8 MB/s ,2503.0 MB/s
     6#silesia_tar       : 211947520 ->  81223047 (2.609),  57.0 MB/s ,2505.5 MB/s
     7#silesia_tar       : 211947520 ->  79381197 (2.670),  42.7 MB/s ,2384.8 MB/s
     8#silesia_tar       : 211947520 ->  78021788 (2.717),  20.5 MB/s ,2424.3 MB/s
     9#silesia_tar       : 211947520 ->  77588039 (2.732),   2.7 MB/s ,2434.7 MB/s
    10#silesia_tar       : 211947520 ->  89799372 (2.360), 169.5 MB/s ,1747.6 MB/s
    11#silesia_tar       : 211947520 ->  85225312 (2.487), 106.5 MB/s ,1672.6 MB/s
    12#silesia_tar       : 211947520 ->  81504376 (2.600),  33.7 MB/s ,1728.2 MB/s
    13#silesia_tar       : 211947520 ->  78536248 (2.699),  22.0 MB/s ,1698.0 MB/s
    14#silesia_tar       : 211947520 ->  75139910 (2.821),  12.8 MB/s ,1653.9 MB/s
    15#silesia_tar       : 211947520 ->  74515732 (2.844),   7.7 MB/s ,1686.4 MB/s
    16#silesia_tar       : 211947520 ->  72841148 (2.910),   4.3 MB/s ,1704.2 MB/s
    17#silesia_tar       : 211947520 ->  70468732 (3.008),   2.9 MB/s ,1658.3 MB/s
    18#silesia_tar       : 211947520 ->  68709617 (3.085),   1.4 MB/s ,1579.6 MB/s
    
    LZ4 v1.7.2 for comparison:
     1#esty\silesia_tar : 211947520 -> 100923164 (47.62%),  437.9 MB/s , 2265.7 MB/s
     2#esty\silesia_tar : 211947520 -> 100923164 (47.62%),  435.2 MB/s , 2254.8 MB/s
     3#esty\silesia_tar : 211947520 ->  82428029 (38.89%),   66.1 MB/s , 2150.8 MB/s
     4#esty\silesia_tar : 211947520 ->  80589674 (38.02%),   54.7 MB/s , 2201.5 MB/s
     5#esty\silesia_tar : 211947520 ->  79427759 (37.48%),   45.1 MB/s , 2209.9 MB/s
     6#esty\silesia_tar : 211947520 ->  78747219 (37.15%),   37.3 MB/s , 2235.3 MB/s
     7#esty\silesia_tar : 211947520 ->  78354189 (36.97%),   31.0 MB/s , 2248.2 MB/s
     8#esty\silesia_tar : 211947520 ->  78140269 (36.87%),   26.3 MB/s , 2239.6 MB/s
     9#esty\silesia_tar : 211947520 ->  78036683 (36.82%),   23.8 MB/s , 2243.9 MB/s

  2. Thanks (6):

    Bulat Ziganshin (7th November 2016),JamesB (8th November 2016),Jyrki Alakuijala (7th November 2016),olokelo (6th May 2019),Simorq (31st May 2017),xcrh (14th November 2016)

  3. #2
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,610
    Thanks
    30
    Thanked 65 Times in 47 Posts
    I suggest having levels 11-19 and 21-29. I think it would make it clearer that these are two different codeds.

  4. #3
    Member
    Join Date
    Nov 2013
    Location
    Kraków, Poland
    Posts
    810
    Thanks
    246
    Thanked 257 Times in 160 Posts
    It's good to have zstd for comparison - from your lzbench:
    Code:
    zstd 1.0.0 -1     244 MB/s     648 MB/s     73659471     34.75
    zstd 1.0.0 -2     186 MB/s     604 MB/s     70168958     33.11
    zstd 1.0.0 -5     90 MB/s     571 MB/s     65002211     30.67
    zstd 1.0.0 -8     31 MB/s     623 MB/s     61026500     28.79
    zstd 1.0.0 -11     16 MB/s     613 MB/s     59523170     28.08
    zstd 1.0.0 -15     5.03 MB/s     638 MB/s     58007776     27.37
    zstd 1.0.0 -18     2.92 MB/s     538 MB/s     55163525     26.03
    zstd 1.0.0 -22     1.55 MB/s     499 MB/s     52772135     24.90
    So your maximum ratio is like level ~9.
    You could get closer 22 without sacrificing speed if using e.g. rans_static instead: ~1500MB/s decoding (~3x faster than FSE) and accurate: https://github.com/jkbonfield/rans_static

  5. #4
    Member
    Join Date
    Dec 2011
    Location
    Cambridge, UK
    Posts
    506
    Thanks
    187
    Thanked 177 Times in 120 Posts
    I've no idea what the compression ratio would be like with rans_static instead. Yes it's more accurate than huffman, but how big a difference it makes depends substantially on the data. I'd be suprised if it bumps it up more than a level or two compared to Zstd. There's also the order-1 variant which is pretty snappy speed still (although don't trust much other than the avx2 version as I still need to do work on order-1 of the other new codecs).

  6. #5
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    309
    Thanks
    68
    Thanked 173 Times in 64 Posts
    Quote Originally Posted by Jarek View Post
    You could get closer 22 without sacrificing speed if using e.g. rans_static instead: ~1500MB/s decoding (~3x faster than FSE) and accurate: https://github.com/jkbonfield/rans_static
    LZ5/LZ5HUF are intented to be between LZ4 and zstd.
    Changing Huffman to ANS will only slightly improve ratio because of 256-symbol alphabet and only 10% of input data is compressed with Huffman.

    Is there rANS_static without SSE and AVX2 that is faster than Yann's Huff0? If there is one I will check it.

  7. #6
    Member
    Join Date
    Dec 2011
    Location
    Cambridge, UK
    Posts
    506
    Thanks
    187
    Thanked 177 Times in 120 Posts
    Quote Originally Posted by inikep View Post
    Is there rANS_static without SSE and AVX2 that is faster than Yann's Huff0? If there is one I will check it.
    No, although the non-SSE variant is still slightly faster decode than FSE (but much slower encode). I do have a variant that icc can auto-vectorise to use AVX2, although it's only icc and it's still not quite on par with the huff0 code when built with gcc (icc rans beats icc huff0 which is just an outlier case).

    Furthermore I've said before that I'd expect tANS to beat rANS if it got the same SIMD treatment. (And obviously Huff0 too.) It's simply history that made me optimise rANS instead. One day I'll get around to experimenting on many-way SIMD tANS.

  8. Thanks:

    inikep (8th November 2016)

  9. #7
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    309
    Thanks
    68
    Thanked 173 Times in 64 Posts
    I released a new version of LZ5 i.e. v2.0 RC2 with improved ratio at https://github.com/inikep/lz5/releases
    LZ5HUF is now LZ5 levels -30 to -49. The results from lzbench:

    Code:
    | Compressor name         | Compression| Decompress.| Compr. size | Ratio |
    | ---------------         | -----------| -----------| ----------- | ----- |
    | memcpy                  |  7332 MB/s |  8719 MB/s |   211947520 |100.00 |
    | lz4 1.7.3               |   440 MB/s |  2318 MB/s |   100880800 | 47.60 |
    | lz4hc 1.7.3 -1          |    98 MB/s |  2121 MB/s |    87591763 | 41.33 |
    | lz4hc 1.7.3 -4          |    55 MB/s |  2259 MB/s |    79807909 | 37.65 |
    | lz4hc 1.7.3 -9          |    22 MB/s |  2315 MB/s |    77892285 | 36.75 |
    | lz4hc 1.7.3 -12         |    17 MB/s |  2323 MB/s |    77849762 | 36.73 |
    | lz4hc 1.7.3 -16         |    10 MB/s |  2323 MB/s |    77841782 | 36.73 |
    | lz5 2.0 RC2 -10         |   346 MB/s |  2610 MB/s |   103402971 | 48.79 |
    | lz5 2.0 RC2 -12         |   103 MB/s |  2458 MB/s |    86232422 | 40.69 |
    | lz5 2.0 RC2 -15         |    50 MB/s |  2552 MB/s |    81187330 | 38.31 |
    | lz5 2.0 RC2 -19         |  3.04 MB/s |  2497 MB/s |    77416400 | 36.53 |
    | lz5 2.0 RC2 -20         |   157 MB/s |  1795 MB/s |    89239174 | 42.10 |
    | lz5 2.0 RC2 -22         |    30 MB/s |  1778 MB/s |    81097176 | 38.26 |
    | lz5 2.0 RC2 -25         |  6.63 MB/s |  1734 MB/s |    74503695 | 35.15 |
    | lz5 2.0 RC2 -29         |  1.37 MB/s |  1634 MB/s |    68694227 | 32.41 |
    | lz5 2.0 RC2 -30         |   246 MB/s |   909 MB/s |    85727429 | 40.45 |
    | lz5 2.0 RC2 -32         |    94 MB/s |  1244 MB/s |    76929454 | 36.30 |
    | lz5 2.0 RC2 -35         |    47 MB/s |  1435 MB/s |    73850400 | 34.84 |
    | lz5 2.0 RC2 -39         |  2.94 MB/s |  1502 MB/s |    69807522 | 32.94 |
    | lz5 2.0 RC2 -40         |   126 MB/s |   961 MB/s |    76100661 | 35.91 |
    | lz5 2.0 RC2 -42         |    28 MB/s |  1101 MB/s |    70955653 | 33.48 |
    | lz5 2.0 RC2 -45         |  6.25 MB/s |  1073 MB/s |    65413061 | 30.86 |
    | lz5 2.0 RC2 -49         |  1.27 MB/s |  1064 MB/s |    60679215 | 28.63 |
    | zlib 1.2.8 -1           |    66 MB/s |   244 MB/s |    77259029 | 36.45 |
    | zlib 1.2.8 -6           |    20 MB/s |   263 MB/s |    68228431 | 32.19 |
    | zlib 1.2.8 -9           |  8.37 MB/s |   266 MB/s |    67644548 | 31.92 |
    | zstd 1.1.1 -1           |   235 MB/s |   645 MB/s |    73659468 | 34.75 |
    | zstd 1.1.1 -2           |   181 MB/s |   600 MB/s |    70168955 | 33.11 |
    | zstd 1.1.1 -5           |    88 MB/s |   565 MB/s |    65002208 | 30.67 |
    | zstd 1.1.1 -8           |    31 MB/s |   619 MB/s |    61026497 | 28.79 |
    | zstd 1.1.1 -11          |    16 MB/s |   613 MB/s |    59523167 | 28.08 |
    | zstd 1.1.1 -15          |  4.97 MB/s |   639 MB/s |    58007773 | 27.37 |
    | zstd 1.1.1 -18          |  2.87 MB/s |   583 MB/s |    55294241 | 26.09 |
    | zstd 1.1.1 -22          |  1.44 MB/s |   505 MB/s |    52731930 | 24.88 |
    | brotli 0.5.2 -0         |   217 MB/s |   244 MB/s |    78226979 | 36.91 |
    | brotli 0.5.2 -2         |    96 MB/s |   283 MB/s |    68066621 | 32.11 |
    | brotli 0.5.2 -5         |    24 MB/s |   312 MB/s |    60801716 | 28.69 |
    | brotli 0.5.2 -8         |  5.56 MB/s |   324 MB/s |    57382470 | 27.07 |
    | brotli 0.5.2 -11        |  0.39 MB/s |   266 MB/s |    51138054 | 24.13 |

  10. Thanks (2):

    comp1 (22nd November 2016),tobijdc (22nd November 2016)

  11. #8
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    309
    Thanks
    68
    Thanked 173 Times in 64 Posts
    LZ5 v2.0 (final) has been released and can be downloaded from https://github.com/inikep/lz5/releases

    LZ5 v2.0 contains 4 compression methods:
    - fastLZ4 : compression levels -10...-19 are designed to give better decompression speed than LZ4 i.e. over 2000 MB/s
    - LZ5v2 : compression levels -20...-29 are designed to give better ratio than LZ4 keeping 75% decompression speed
    - fastLZ4 + Huffman : compression levels -30...-39 add Huffman coding to fastLZ4
    - LZ5v2 + Huffman : compression levels -40...-49 give the best ratio (comparable to zlib and low levels of zstd/brotli) at decompression speed of 1000 MB/s

    In comparison to LZ5 v2.0 RC2 it has new and faster levels 20 and 40.
    Last edited by inikep; 6th February 2017 at 00:53.

  12. Thanks (5):

    Bulat Ziganshin (10th February 2017),encode (6th February 2017),Jyrki Alakuijala (6th February 2017),milky (6th February 2017),spark (6th February 2017)

  13. #9
    Member
    Join Date
    May 2012
    Location
    Germany
    Posts
    39
    Thanks
    25
    Thanked 105 Times in 24 Posts
    Which method of them is the same, that in LZ5 1.5 was used?

  14. #10
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    309
    Thanks
    68
    Thanked 173 Times in 64 Posts
    Quote Originally Posted by milky View Post
    Which method of them is the same, that in LZ5 1.5 was used?
    None. LZ5 v2.0 is completely rewritten with new codewords and a new file format. I was thinking about adding LZ5v1 but I decided it was too confusing and v2.0 was superior to all compression levels of v1.5.

  15. #11
    Member
    Join Date
    May 2012
    Location
    Germany
    Posts
    39
    Thanks
    25
    Thanked 105 Times in 24 Posts
    I am asking, cause I wanted to add it to 7-Zip ZS ... what should I do with it now

    - "7z a archive.7z -m0=lz5 -mmt=4 some-files" will compress with version 1.5

    Should I take these additional for the 4 new methods:
    - "-mx0..9 -m0=fastLZ4" (LZ5v2 Level 10-19)
    - "-mx0..9 -m0=LZ5v2" (LZ5v2 Level 20-29)
    - "-mx0..9 -m0=fastLZ4Huff" (LZ5v2 Level 30-39)
    - "-mx0..9 -m0=LZ5v2Huff" (LZ5v2 Level 40-49)

    What do you think about it... or should I just leave it with LZ5 v1.5 ... cause LZ5 is only used for experimenting ?

  16. #12
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    4,008
    Thanks
    301
    Thanked 1,324 Times in 757 Posts
    You can actually specify -m0=lz5:x40 ...

  17. #13
    Member
    Join Date
    May 2012
    Location
    Germany
    Posts
    39
    Thanks
    25
    Thanked 105 Times in 24 Posts
    Yes, but LZ5 v1.5 got 16 Levels I think .. so what should happen with them ?

  18. #14
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    309
    Thanks
    68
    Thanked 173 Times in 64 Posts
    Quote Originally Posted by milky View Post
    Yes, but LZ5 v1.5 got 16 Levels I think .. so what should happen with them ?
    You can use -m0=lz5v2:x40 but it think treating LZ5 v2.0 as 4 independent codecs makes more sense.

    One more thing. I expect that when you will try to have both v1.5 and v2.0 in a single executable there can be namespace collision which can be solved preferably by renaming all "LZ5" strings with "LZ5v1" for v1.5.

    I assume that LZ5 v2.0 is stable. It passes fuzzer tests what was not tested for v1.5. I will not change file format in v2.x.

  19. #15
    Member
    Join Date
    May 2012
    Location
    Germany
    Posts
    39
    Thanks
    25
    Thanked 105 Times in 24 Posts
    I will think about it. When Igor releases a new version, I will also update LZ5 next time

    The namespace collisions will be no problem, as you suggested, I will do some renaming, when needed.

  20. Thanks:

    inikep (7th February 2017)

  21. #16
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,576
    Thanks
    790
    Thanked 687 Times in 372 Posts
    this line seems a copypaste error

  22. #17
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    309
    Thanks
    68
    Thanked 173 Times in 64 Posts
    Quote Originally Posted by Bulat Ziganshin View Post
    this line seems a copypaste error
    Actually it's not. All files taken from LZ4 contain Yann's copyright. I added my name only to files where I changed something important. The files where only "LZ4" is replaced to "LZ5" are left without my name.

  23. #18
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,576
    Thanks
    790
    Thanked 687 Times in 372 Posts
    yeah, these lines are just a comment, but i understood it as "LZ5 - Fast LZ compression algorithm. Copyright (C) 2011-2014, Yann Collet". it's not obvious that this line tells only about this concrete file

  24. #19
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    918
    Thanks
    254
    Thanked 327 Times in 200 Posts
    Would it be a good idea to add LZ5 into squashfs? Does it compress better than LZ4 for small segments (something like 4-128 kB) of data?

  25. #20
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    309
    Thanks
    68
    Thanked 173 Times in 64 Posts
    I think that LZ5 should be positioned between LZ4 and zstd (at low levels) in terms of ratio and decompression speed. The following results comes from lzbench.exe -jr -elz4/lz4hc/lz5 with 354 files smaller or equal to 64 KB:
    Code:
    lz4 1.7.5                 461 MB/s  1919 MB/s     3616250  41.32 354 files
    lz4hc 1.7.5 -1            112 MB/s  2074 MB/s     3259798  37.25 354 files
    lz4hc 1.7.5 -2            103 MB/s  2122 MB/s     3169953  36.22 354 files
    lz4hc 1.7.5 -3             91 MB/s  2152 MB/s     3117906  35.62 354 files
    lz4hc 1.7.5 -4             80 MB/s  2176 MB/s     3088475  35.29 354 files
    lz4hc 1.7.5 -5             71 MB/s  2186 MB/s     3071615  35.10 354 files
    lz4hc 1.7.5 -6             63 MB/s  2192 MB/s     3061388  34.98 354 files
    lz4hc 1.7.5 -7             53 MB/s  2198 MB/s     3055612  34.91 354 files
    lz4hc 1.7.5 -8             45 MB/s  2198 MB/s     3052299  34.87 354 files
    lz4hc 1.7.5 -9             40 MB/s  2203 MB/s     3050894  34.86 354 files
    lz4hc 1.7.5 -10            24 MB/s  2205 MB/s     3050071  34.85 354 files
    lz4hc 1.7.5 -11          8.30 MB/s  2213 MB/s     3036799  34.70 354 files
    lz4hc 1.7.5 -12          1.04 MB/s  2200 MB/s     3034573  34.67 354 files
    lz5 2.0 -30               226 MB/s  1041 MB/s     3202538  36.59 354 files
    lz5 2.0 -31                49 MB/s  1180 MB/s     3209725  36.67 354 files
    lz5 2.0 -32               109 MB/s  1153 MB/s     2980730  34.06 354 files
    lz5 2.0 -33                38 MB/s  1288 MB/s     2997320  34.25 354 files
    lz5 2.0 -34                30 MB/s  1336 MB/s     2951511  33.72 354 files
    lz5 2.0 -35                28 MB/s  1388 MB/s     2932114  33.50 354 files
    lz5 2.0 -36                27 MB/s  1458 MB/s     2932694  33.51 354 files
    lz5 2.0 -37                25 MB/s  1542 MB/s     2907824  33.22 354 files
    lz5 2.0 -38                20 MB/s  1599 MB/s     2897059  33.10 354 files
    lz5 2.0 -39              1.02 MB/s  1379 MB/s     2860620  32.68 354 files
    lz5 2.0 -40               177 MB/s   808 MB/s     3120284  35.65 354 files
    lz5 2.0 -41              2.72 MB/s   887 MB/s     2915562  33.31 354 files
    lz5 2.0 -42              2.52 MB/s   889 MB/s     2900656  33.14 354 files
    lz5 2.0 -43              2.40 MB/s   951 MB/s     2825415  32.28 354 files
    lz5 2.0 -44              0.88 MB/s   966 MB/s     2789945  31.88 354 files
    lz5 2.0 -45              0.85 MB/s  1031 MB/s     2783172  31.80 354 files
    lz5 2.0 -46              0.83 MB/s   921 MB/s     2751897  31.44 354 files
    lz5 2.0 -47              0.62 MB/s   980 MB/s     2723447  31.12 354 files
    lz5 2.0 -48              0.61 MB/s   994 MB/s     2712381  30.99 354 files
    lz5 2.0 -49              0.26 MB/s   996 MB/s     2705798  30.92 354 files
    zlib 1.2.8 -1              19 MB/s   260 MB/s     2708921  30.95 354 files
    zlib 1.2.8 -2              76 MB/s   266 MB/s     2638522  30.15 354 files
    zlib 1.2.8 -3              67 MB/s   271 MB/s     2587869  29.57 354 files
    zlib 1.2.8 -4              57 MB/s   267 MB/s     2483155  28.37 354 files
    zlib 1.2.8 -5              42 MB/s   270 MB/s     2416030  27.61 354 files
    zlib 1.2.8 -6              31 MB/s   274 MB/s     2394064  27.35 354 files
    zlib 1.2.8 -7              25 MB/s   275 MB/s     2387677  27.28 354 files
    zlib 1.2.8 -8              15 MB/s   276 MB/s     2381969  27.22 354 files
    zlib 1.2.8 -9              12 MB/s   276 MB/s     2380786  27.20 354 files
    zstd 1.1.3 -1             195 MB/s   557 MB/s     2557413  29.22 354 files
    zstd 1.1.3 -22           3.39 MB/s   545 MB/s     2231497  25.50 354 files
    brotli 0.5.2 -0           191 MB/s   182 MB/s     2817875  32.20 354 files
    brotli 0.5.2 -1           170 MB/s   188 MB/s     2640228  30.17 354 files
    brotli 0.5.2 -11         0.51 MB/s   225 MB/s     2018170  23.06 354 files
    Last edited by inikep; 10th February 2017 at 17:48.

  26. Thanks:

    Jyrki Alakuijala (10th February 2017)

  27. #21
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    918
    Thanks
    254
    Thanked 327 Times in 200 Posts
    Impressive results. Would you add gzip and high density results, I.e., zstd 22 and brotli 11. Often squashfs use is read only and compression time doesn't matter much.

  28. #22
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    309
    Thanks
    68
    Thanked 173 Times in 64 Posts
    Quote Originally Posted by Jyrki Alakuijala View Post
    Impressive results. Would you add gzip and high density results, I.e., zstd 22 and brotli 11. Often squashfs use is read only and compression time doesn't matter much.
    I added new results. The decompression speed is great, but compression speed for levels 41-49 (the same as 31-39 but with Huffman coding) is not good for small files.

  29. #23
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,576
    Thanks
    790
    Thanked 687 Times in 372 Posts
    i started adding LZ5 to freearc. one thing i encountered is that __builtin_bswap32/64 were added only in gcc 4.3, so you may replace the checks:
    Code:
    #elif (LZ5_GCC_VERSION >= 403)
        return __builtin_bswap32(in);
    ...
    #elif (LZ5_GCC_VERSION >= 403)
        return __builtin_bswap64(in);
    this problem also beats zstd and probably huff0, and it's the only caveat for compilation of zstd by gcc 3.4.5 - great!

    EDIT: LZ5 also successfully compiled with gcc 3.4.5. I don't yet checked whether this compilation of zstd/lz5 works correct
    Last edited by Bulat Ziganshin; 12th February 2017 at 18:12.

  30. #24
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,576
    Thanks
    790
    Thanked 687 Times in 372 Posts
    i use amalgamated compilation with the following code:
    Code:
    extern "C" {
    #define HUF_STATIC_LINKING_ONLY  /* HUF_TABLELOG_ABSOLUTEMAX */
    #include "lib\lz5_compress.h"
    #include "lib\lz5_decompress.h"
    #include "lib\lz5_compress.c"
    #include "lib\lz5_decompress.c"
    #include "lib\entropy\entropy_common.c"
    #include "lib\entropy\fse_compress.c"
    #include "lib\entropy\fse_decompress.c"
    #include "lib\entropy\huf_compress.c"
    #include "lib\entropy\huf_decompress.c"
    #include "lib\xxhash\xxhash.c"
    }
    so far i have the following:


    Code:
    cl /c C_LZ5.cpp /TP -Gy -GR- -O2 -EHsc /DFREEARC_WIN /DFREEARC_INTEL_BYTE_ORDER /DUNICODE /D_UNICODE
    
    
    c:\!freearc\freearc\compression\lz5\lib\lz5_compress.c(305): warning C4551: function call missing argument list
    c:\!freearc\freearc\compression\lz5\lib\lz5_compress.c(306): warning C4551: function call missing argument list
    c:\!freearc\freearc\compression\lz5\lib\lz5_decompress_lz4.h(36): warning C4551: function call missing argument list
    c:\!freearc\freearc\compression\lz5\lib\lz5_decompress_lz5v2.h(45): warning C4551: function call missing argument list
    c:\!freearc\freearc\compression\lz5\lib\lz5_decompress.c(291): warning C4551: function call missing argument list
    c:\!freearc\freearc\compression\lz5\lib\entropy\fse_compress.c(196): error C2084: function 'short FSE_abs(short)' already has a body
    c:\!freearc\freearc\compression\lz5\lib\entropy\entropy_common.c(65): note: see previous definition of 'FSE_abs'
    c:\!freearc\freearc\compression\lz5\lib\entropy\fse_compress.c(255): error C2568: '-=': unable to resolve function overload
    c:\!freearc\freearc\compression\lz5\lib\entropy\fse_compress.c(255): note: could be 'short FSE_abs(short)'
    C
    FSE_abs() defined in different modules is my usual problem with Cyan libs but all those "(void)LZ5_wildCopy;" looks pretty strange and makes msvc mad. what's the their meaning?

  31. #25
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,576
    Thanks
    790
    Thanked 687 Times in 372 Posts
    lz5 2.0 -40 177 MB/s 808 MB/s 3120284 35.65 354 files
    lz5 2.0 -41 2.72 MB/s 887 MB/s 2915562 33.31 354 files
    lz5 2.0 -42 2.52 MB/s 889 MB/s 2900656 33.14 354 files
    it seems that lz5 has too large dictionary for these files and spends most of time initializing hash structure. if it will be able to reduce dictionary to the next power of 2 after filesize, speed should be pretty close to that on large files (as -40 already does)

  32. #26
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    309
    Thanks
    68
    Thanked 173 Times in 64 Posts
    With https://github.com/inikep/lz5/commit...3c6c43145e2194 I fixed:
    a) __builtin_bswap32/64 with gcc older than 4.3
    b) "(void)LZ5_wildCopy;" - this was for warnings with "-Wunused-functions" but seems not needed anymore

    I will look at compression speed with small files when I will have some spare time.

  33. #27
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,576
    Thanks
    790
    Thanked 687 Times in 372 Posts
    the small "inefficiency" in your/Cyan code: according to standard, undefined identifiers are evaluated to 0:

    http://en.cppreference.com/w/c/preprocessor/conditional :
    #if, #elif

    The expression is a constant expression, using only literals and identifiers, defined using #define directive. Any identifier, which is not literal, non defined using #define directive, evaluates to 0.
    so these checks, as well as lot of similar ones, may be simplified to

    #elif __GNUC__ * 100 + __GNUC_MINOR__ >= 403

  34. #28
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    309
    Thanks
    68
    Thanked 173 Times in 64 Posts
    Quote Originally Posted by Bulat Ziganshin View Post
    the small "inefficiency" in your/Cyan code: according to standard, undefined identifiers are evaluated to 0
    The issue here is gcc with -Wall -Wextra which throws warnings:
    ../lib/common/xxhash.c:207:7: warning: "GCC_VERSSION" is not defined [-Wundef]

  35. #29
    Member
    Join Date
    Jul 2013
    Location
    United States
    Posts
    194
    Thanks
    44
    Thanked 140 Times in 69 Posts
    Quote Originally Posted by Bulat Ziganshin View Post
    i started adding LZ5 to freearc. one thing i encountered is that __builtin_bswap32/64 were added only in gcc 4.3, so you may replace the checks:
    I threw together some endian-handling code a while back, it may be useful:

    https://github.com/nemequ/portable-s...ndian/endian.h

    It's public domain, and it's the most complete endianness code I'm aware of… really the only thing missing is support for PDP endian.

    Quote Originally Posted by Bulat Ziganshin View Post
    so these checks, as well as lot of similar ones, may be simplified to

    #elif __GNUC__ * 100 + __GNUC_MINOR__ >= 403
    Hedley has some macros which may help there… I'm not sure what exactly you're doing with those, but the HEDLEY_GCC_HAS_*() macros tend to be very useful instead of just relying on GCC version; they will also check clang using the __has_*() macros, so the results should be more accurate. clang tends to add features without bumping __GNUC__ and friends.

  36. Thanks (2):

    Cyan (13th February 2017),inikep (13th February 2017)

  37. #30
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,576
    Thanks
    790
    Thanked 687 Times in 372 Posts
    https://github.com/Cyan4973/FiniteSt...ropy/issues/77 is also applicable to LZ5

    Quote Originally Posted by inikep View Post
    The issue here is gcc with -Wall -Wextra which throws warnings:
    ../lib/common/xxhash.c:207:7: warning: "GCC_VERSSION" is not defined [-Wundef]
    nice. there is always room to learn new things

    Quote Originally Posted by nemequ View Post
    I threw together some endian-handling code a while back, it may be useful:

    https://github.com/nemequ/portable-s...ndian/endian.h

    It's public domain, and it's the most complete endianness code I'm aware of… really the only thing missing is support for PDP endian.
    nice too. i love small libs

Page 1 of 2 12 LastLast

Similar Threads

  1. LZ5 v2.0 beta
    By inikep in forum Data Compression
    Replies: 4
    Last Post: 4th November 2016, 17:22
  2. FYI: Doboz - fast decompression
    By thometal in forum Data Compression
    Replies: 3
    Last Post: 8th January 2014, 22:47
  3. Very slow compression with very fast decompression?
    By Mangix in forum Data Compression
    Replies: 16
    Last Post: 12th September 2013, 03:25
  4. Most efficient/practical compression method for short strings?
    By never frog in forum Data Compression
    Replies: 6
    Last Post: 1st September 2009, 05:05
  5. Fast decompression of big files
    By SvenBent in forum Forum Archive
    Replies: 16
    Last Post: 8th March 2008, 20:17

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •