Page 1 of 14 12311 ... LastLast
Results 1 to 30 of 402

Thread: Zstandard

  1. #1
    Member
    Join Date
    Sep 2008
    Location
    France
    Posts
    870
    Thanks
    471
    Thanked 264 Times in 109 Posts

    Zstandard

    Just for information :

    I've recently released a first version of Zstd,
    the open source successor of zhuff.

    It's not finished yet, but this version seems in good enough shape for release and public comments.
    You can download its source code on github.


    While the decoding speed seems mostly settled,
    the compression side, on the other hand, still requires a lot of work.
    Only the "fast mode" is released so far.

    An "HC" version can be created. It will require quite some time and expertise to reach its conclusion.

    Of course, all comments and contributions are very welcomed.
    Last edited by Cyan; 24th October 2015 at 23:24.

  2. Thanks (18):

    avitar (2nd February 2015),Bulat Ziganshin (25th January 2015),compgt (17th May 2019),encode (18th November 2016),Gonzalo (25th January 2015),Jarek (25th January 2015),jethro (26th January 2015),kassane (19th January 2016),lorents17 (31st August 2015),Mat Chartier (11th February 2015),Matt Mahoney (25th January 2015),milky (6th February 2017),m^2 (25th January 2015),Nania Francesco (26th January 2015),Razor12911 (22nd July 2016),samsat1024 (28th September 2016),sathex (21st August 2019),Stephan Busch (25th January 2015)

  3. #2
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,537
    Thanks
    758
    Thanked 676 Times in 366 Posts
    1. it's the most awkward name i've seen in compression
    2. without repdist/repboth codes it will have less max. compression ratio than rar/tornado
    3. Core i5-4300U has frequency up to 2.90 GHz, and i guess that it was actual freq in the test. you can use sensors window of hwinfo to check actual frequency during (de)compression
    4. can you release the binary so i can check it here?
    Last edited by Bulat Ziganshin; 25th January 2015 at 14:17.

  4. #3
    Member
    Join Date
    Sep 2008
    Location
    France
    Posts
    870
    Thanks
    471
    Thanked 264 Times in 109 Posts
    Zstd includes a repDist code.
    But the current version only takes passively advantage of it, rather than actively looking for it.
    It's intended to be more useful at stronger compression modes (which still have to be developed).

    Please find in attached file a binary version compiled using Visual 2012.
    Attached Files Attached Files

  5. Thanks:

    Nania Francesco (26th January 2015)

  6. #4
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Yann, I see that you use fsbench. I'd love to get even a rough patch, so I can incorporate it in mine version faster.

  7. #5
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,537
    Thanks
    758
    Thanked 676 Times in 366 Posts
    1. yes, you should set the compression format first, and then more backends will arrive. but in order to select the best compression format, you need to have several backends - from lz4-like one to optimum-parsing in order to check each decision in various compression domains. i may look into porting OP backend so that you can see the compression limits of the format

    2. tornado includes 4 repdist codes (like rar/lzma), repboth code (len+dist) and funny "rep_dist1 + len:=1" code. also it includes table2/table4 codes that just diffs sequential 2/4-byte words prior to compression (tor.exe -t0 option disables table preprocessing, so you can check its effect). if you think that these features may slow down the decoder, you may provide multiple decoder levels - fastest one without (extra) rep codes. afair, repdist2..4 codes improved compression by 1..2%

    3. tornado greedy/lazy compressor also has only passive repdist recognition, tornado OP as well as both lzma modes has active repdist recognition

  8. Thanks:

    Cyan (2nd November 2015)

  9. #6
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,257
    Thanks
    307
    Thanked 795 Times in 488 Posts
    I updated LTCB and Silesia. On LTCB it makes the Pareto frontier for compression speed. It didn't for decompression probably because crush was run on a faster machine.
    http://mattmahoney.net/dc/text.html#3546
    http://mattmahoney.net/dc/silesia.html

    Edit: for some reason, enwik9.zst was 1 byte bigger in the Windows test. (Decompression was OK in both tests).
    Last edited by Matt Mahoney; 25th January 2015 at 22:48.

  10. #7
    Member
    Join Date
    Aug 2014
    Location
    Argentina
    Posts
    536
    Thanks
    236
    Thanked 90 Times in 70 Posts
    Frankly, I am impressed. Very good job for this very first version. I was looking for a good replacement for slugx, and finally I think I've found one.

    Just informing you, compiling and compressing fine on Linux x86

    BTW... I think most readers will understand your program's name as the 'Ultimate... well, STD'
    Last edited by Gonzalo; 26th January 2015 at 00:52.

  11. #8
    Member
    Join Date
    Aug 2014
    Location
    Argentina
    Posts
    536
    Thanks
    236
    Thanked 90 Times in 70 Posts
    Error:
    Read : 700 MB ==> 50.22% *** stack smashing detected ***: zstd terminated
    Command: wine shar as - WINDOWS | wine fazip rep:400m - - | time zstd - WINDOWS.compress.

    Are you trying to get root on my machine???
    Last edited by Gonzalo; 26th January 2015 at 01:44.

  12. #9
    Member
    Join Date
    Aug 2014
    Location
    Argentina
    Posts
    536
    Thanks
    236
    Thanked 90 Times in 70 Posts
    Another one:

    Code:
    time wine slugx d WINDOWS.pack : | zstd - WINDOWS.comp
    
    -------------------------------------------------------------------------------
     Slug X (Apr  6 2009) - Copyright (c) 2009 by Christian Martelock
    -------------------------------------------------------------------------------
    
     Decompressing...
     src:    'WINDOWS.pack'
     dst:    ':'
    
    Read : 1446 MB  ==> 52.21%   Error 24 : Compression error : ZSTD_ERROR_GENERIC 
     1010001.91 KB -> -0.00 KB (-0.00%, -1 bytes)
    
    real    4m32.717s
    user    3m12.644s
    sys    0m14.488s
    Apparently, both are related to memory pipeline. Perhaps is wine's fault. File-to-file jobs are correct. Also, "zstd -d in | zstd - out" performs correctly as well.
    Last edited by Gonzalo; 26th January 2015 at 02:45.

  13. #10
    Member
    Join Date
    Aug 2013
    Location
    France
    Posts
    77
    Thanks
    27
    Thanked 26 Times in 11 Posts
    Very nice work !!

    Here is an updated version of the efficiency benchmark I did a while ago, now including zstd .

    Benchmark
    Performed on a Macbook Pro, OSX 10.10.1, 2.3 GHz Intel Core i7, 8Go 1600 MHz DDR, SSD
    All programs were compiled with their standard options (standard makefile) using clang (note that I also tried with gcc-4.9 and it makes no difference whatsoever). They were run a minimum of 5 times and chronometered using the best "user" value from the "time" utility. They are of course all single-threaded otherwise this would make no sense.
    The file used was the world famous enwik8 but any compressible data would do !

    Compression

    efficiency = (1/(compression_ratio ^ power))/compression_time

    Click image for larger version. 

Name:	compression_efficiency.png 
Views:	450 
Size:	66.9 KB 
ID:	3371

    X-axis : power (higher powers means ratio is considered more important than speed).
    Y-axis : percentage of total (higher is better)


    zstd is the most efficient when one thinks the ratio is a power of 5 at least more important than speed.
    Below that sharc/density is clearly the most efficient.

    Decompression

    efficiency = (1/(compression_ratio ^ power))/decompression_time

    Click image for larger version. 

Name:	decompression_efficiency.png 
Views:	363 
Size:	67.5 KB 
ID:	3372

    X-axis : power (higher powers means ratio is considered more important than speed).
    Y-axis : percentage of total (higher is better)

    lz4 is ahead on decompression efficiency, but it's not really a surprise as it was optimized for that purpose.
    lz4 -1, lz4 -9 and sharc -c1 decompress at roughly the same speeds but lz4 -9 's ratio is much better hence the results.

    Round trip

    This is usually the most interesting as it shows the general algorithm efficiency.

    efficiency = (1/(compression_ratio ^ power))/roundtrip_time

    Click image for larger version. 

Name:	roundtrip_efficiency.png 
Views:	355 
Size:	67.2 KB 
ID:	3373

    X-axis : power (higher powers means ratio is considered more important than speed).
    Y-axis : percentage of total (higher is better)

    sharc/density is ahead when one considers that ratio is less than a power of 4 more important than speed.
    After that zstd is ahead because of a better ratio.

    PS. it would be nice if you could add the sharc/density results in your benchmark on your zstd github repository Cyan .
    Last edited by gpnuma; 26th January 2015 at 03:55.

  14. #11
    Member
    Join Date
    Sep 2008
    Location
    France
    Posts
    870
    Thanks
    471
    Thanked 264 Times in 109 Posts
    > Yann, I see that you use fsbench. I'd love to get even a rough patch, so I can incorporate it in mine version faster.

    @m^2 : I had to make a lot of little changes to add zstd.
    I don't know how I can retrieve them to provide them to you as a compatible patch with your version of fsbench.

    Maybe as an easier intermediate step, I could send you the full source code of the current version I use ? (basically, v0.14.3 + zstd modifications)

    On a longer term, I would like to suggest refactoring the code so that adding a codec could be achieved with some localized modifications (as opposed to spread into multiple files). But I do acknowledge it is probably a significant work.


    > 1. yes, you should set the compression format first, and then more backends will arrive. but in order to select the best compression format, you need to have several backends - from lz4-like one to optimum-parsing in order to check each decision in various compression domains. i may look into porting OP backend so that you can see the compression limits of the format

    @Bulat : You are totally right.
    This is the current work ongoing.
    It will be great thing if you can achieve such a port, I trust a lot your skills at high compression parsing.

    As a sidenote, maybe I should modify the current compression code a bit, for a clearer separation between match-selection (parsing) and encoding.
    This would allow to include multiple match/parsers reusing a common encoding code.
    It's not that difficult, since it's almost already done. It probably just need a clearer/easier interface contract.


    @matt : Thanks for testing !

    > Edit: for some reason, enwik9.zst was 1 byte bigger in the Windows test. (Decompression was OK in both tests).

    The 1 byte difference between Linux & Windows, even though harmless, is nonetheless unexpected.
    I'll try to reproduce it to understand what's going on.

    Sidenote : I intend to provide a Visual project directory, for people who prefer this IDE. Will it be useful to you ?


    @Gonzalo :

    > Apparently, both are related to memory pipeline. Perhaps is wine's fault. File-to-file jobs are correct. Also, "zstd -d in | zstd - out" performs correctly as well.

    Thanks for reporting
    So, my understanding is that you used the Visual compiled windows binary in combination with Wine.
    It's indeed a combination that was never tested before.
    Maybe you found a bug on the pipe implementation.
    It's unclear if it can be solved from source code side. Worth investigating.

    In any case, if you are using a Linux machine, maybe compiling with a native Linux compiler (gcc, clang, etc.) will help you complete your tests.


    @gpnuma :
    Thanks for the kind words.
    Your findings are very instructive.

    > PS. it would be nice if you could add the sharc/density results in your benchmark on your zstd github repository Cyan .

    Sure. As you may know, I'm currently using m^2's fsbench for testings.
    fsbench doesn't yet integrate density, which is the reason why it's not yet present in the comparison.
    This can probably be improved, I may add it on my own, or even switch to a different test code able to produce the comparison exercise.

    Note that I tend to prefer the Silesia Corpus as a reference.
    Enwik is a famous reference, rightly so thanks to Matt,
    but it also stress a well-defined workload (text files), which, alone, is not representative enough of the various inputs a codec is expected to deal with.

  15. #12
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by Cyan View Post
    > Yann, I see that you use fsbench. I'd love to get even a rough patch, so I can incorporate it in mine version faster.

    @m^2 : I had to make a lot of little changes to add zstd.
    I don't know how I can retrieve them to provide them to you as a compatible patch with your version of fsbench.

    Maybe as an easier intermediate step, I could send you the full source code of the current version I use ? (basically, v0.14.3 + zstd modifications)

    On a longer term, I would like to suggest refactoring the code so that adding a codec could be achieved with some localized modifications (as opposed to spread into multiple files). But I do acknowledge it is probably a significant work.
    There are indeed many steps involved in adding a new codec, though they are documented in codecs.hpp. Did you see it? I doced it for my own needs, but a README might be a better place.
    Putting all changes in a single place is not something I tried to do. I will thing about it.

    I'll gladly take the source.

    @gpnuma :
    Thanks for the kind words.
    Your findings are very instructive.

    > PS. it would be nice if you could add the sharc/density results in your benchmark on your zstd github repository Cyan .

    Sure. As you may know, I'm currently using m^2's fsbench for testings.
    fsbench doesn't yet integrate density, which is the reason why it's not yet present in the comparison.
    This can probably be improved, I may add it on my own, or even switch to a different test code able to produce the comparison exercise.

    Note that I tend to prefer the Silesia Corpus as a reference.
    Enwik is a famous reference, rightly so thanks to Matt,
    but it also stress a well-defined workload (text files), which, alone, is not representative enough of the various inputs a codec is expected to deal with.
    I tried to add libssc to fsbench, harder than I would usually because of reported performance, but couldn't make it work, the code just seemed broken. I noticed that gpnuma made some updates recently and intend to try again, but I have neither much time nor much faith in success, so zstd is a higher priority task for me.
    Last edited by m^2; 26th January 2015 at 18:23.

  16. #13
    Member
    Join Date
    Aug 2014
    Location
    Argentina
    Posts
    536
    Thanks
    236
    Thanked 90 Times in 70 Posts
    So, my understanding is that you used the Visual compiled windows binary in combination with Wine.
    Nope. I compiled myself Zstd using 'make install' as is the source and MAKEFILE in the package. That means I'm using a proper Linux ELF. There are also some windows programs throwing data to it via pipeline.
    Windows programs are Shelwien's shar, Bulat's fazip and Christian's slugx, each one on different runs to make me sure.
    Attached Files Attached Files
    Last edited by Gonzalo; 27th January 2015 at 01:51.

  17. #14
    Member
    Join Date
    Aug 2013
    Location
    France
    Posts
    77
    Thanks
    27
    Thanked 26 Times in 11 Posts
    Quote Originally Posted by m^2 View Post
    harder than I would usually because of reported performance, but couldn't make it work, the code just seemed broken. I noticed that gpnuma made some updates recently and intend to try again, but I have neither much time nor much faith in success, so zstd is a higher priority task for me.
    Lol you're right the master branch was broken at the time that's why I wanted you to checkout the latest beta release instead

    IMO what you should do m^2 is publish your code on github (it's free) !
    I will then create all the modifications necessary in a fork and send you a pull request , this would actually be the simplest for everyone to add a codec to your benchmark hence insuring its well deserved success !

  18. #15
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Beta was broken the same.
    I prefer chiselapp:
    https://chiselapp.com/user/Justin_be...itory/fsbench/

    it serves the same purpose and unlike git, fossil is free from copyleft.

  19. #16
    Member
    Join Date
    Sep 2008
    Location
    France
    Posts
    870
    Thanks
    471
    Thanked 264 Times in 109 Posts
    @m^2 : Please find in attached file
    the modified source directory of fsbench 0.14.3
    including the modifications to integrate Zstd.

    Hope it can help
    Attached Files Attached Files
    Last edited by Cyan; 31st January 2015 at 14:34.

  20. Thanks:

    m^2 (27th January 2015)

  21. #17
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    ZSTD support is added it fsbench.
    Clang 3.3 -O3, Phenom 2@3.2 ghz:
    Code:
    m% ./fsbench fast miniz,1 zlib,1 /mnt/usr/home/m/bench/scc1.tar
    Codec                                   version      args
    C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
    lrrle                                   0            256
       12281040 (x 1.026)     3069 MB/s 3087 MB/s        76e6   76e6
    LZ4                                     r114         
        7201745 (x 1.749)      328 MB/s 1139 MB/s       140e6  487e6
    LZF                                     3.6          very
        7090337 (x 1.776)      176 MB/s  507 MB/s        76e6  221e6
    LZO                                     2.06         1x1
        7152460 (x 1.761)      383 MB/s  601 MB/s       165e6  259e6
    QuickLZ                                 1.5.1b6      1
        6804291 (x 1.851)      336 MB/s  327 MB/s       154e6  150e6
    Snappy                                  1.1.0        
        7070230 (x 1.781)      322 MB/s  745 MB/s       141e6  326e6
    ZSTD                                    0.0.1        
        5510621 (x 2.286)      141 MB/s  405 MB/s        79e6  227e6
    miniz                                   1.11         1
        6245122 (x 2.017)      102 MB/s  180 MB/s        51e6   90e6
    zlib                                    1.2.8        1
        5583512 (x 2.256)     40.9 MB/s  223 MB/s        22e6  124e6
    Codec                                   version      args
    C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
    done... (4*X*1) iteration(s)).
    
    m% ./fsbench -b131072 -m4096 fast miniz,1 zlib,1 /mnt/usr/home/m/bench/scc1.tar
    Codec                                   version      args
    C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
    lrrle                                   0            256
       12328960 (x 1.022)     3009 MB/s 5317 MB/s        63e6  112e6
    LZ4                                     r114         
        7503872 (x 1.678)      325 MB/s 1130 MB/s       131e6  447e6
    LZF                                     3.6          very
        7303168 (x 1.725)      170 MB/s  496 MB/s        71e6  205e6
    LZO                                     2.06         1x1
        7372800 (x 1.708)      378 MB/s  537 MB/s       156e6  217e6
    QuickLZ                                 1.5.1b6      1
        7073792 (x 1.781)      339 MB/s  330 MB/s       148e6  142e6
    Snappy                                  1.1.0        
        7266304 (x 1.733)      321 MB/s  728 MB/s       135e6  304e6
    ZSTD                                    0.0.1        
    Decoding failed!%            
                    m% ./fsbench -b4096 -m512 fast miniz,1 zlib,1 /mnt/usr/home/m/bench/scc1.tar
    Codec                                   version      args
    C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
    lrrle                                   0            256
       12382208 (x 1.017)     2765 MB/s 4691 MB/s        46e6   79e6
    LZ4                                     r114         
        9083904 (x 1.387)      320 MB/s 1111 MB/s        89e6  306e6
    LZF                                     3.6          very
        8790016 (x 1.433)      154 MB/s  472 MB/s        46e6  142e6
    LZO                                     2.06         1x1
        8924160 (x 1.411)      349 MB/s  567 MB/s       101e6  160e6
    QuickLZ                                 1.5.1b6      1
        8901632 (x 1.415)      240 MB/s  319 MB/s        70e6   88e6
    Snappy                                  1.1.0        
        8878080 (x 1.419)      329 MB/s  726 MB/s        97e6  209e6
    ZSTD                                    0.0.1        
    Decoding failed!%    
                         m% ./fsbench -b131072 fast miniz,1 zlib,1 /mnt/usr/home/m/bench/scc1.tar 
    Codec                                   version      args
    C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
    lrrle                                   0            256
       12281040 (x 1.026)     3021 MB/s 3315 MB/s        75e6   82e6
    LZ4                                     r114         
        7267513 (x 1.733)      326 MB/s 1144 MB/s       137e6  483e6
    LZF                                     3.6          very
        7119193 (x 1.769)      169 MB/s  500 MB/s        73e6  211e6
    LZO                                     2.06         1x1
        7175287 (x 1.755)      377 MB/s  557 MB/s       162e6  234e6
    QuickLZ                                 1.5.1b6      1
        6864057 (x 1.835)      338 MB/s  331 MB/s       153e6  148e6
    Snappy                                  1.1.0        
        7070516 (x 1.781)      320 MB/s  739 MB/s       140e6  324e6
    ZSTD                                    0.0.1        
        5885924 (x 2.140)      151 MB/s  410 MB/s        80e6  213e6
    miniz                                   1.11         1
        6254371 (x 2.014)      100 MB/s  180 MB/s        50e6   90e6
    zlib                                    1.2.8        1
        5628481 (x 2.238)     43.5 MB/s  218 MB/s        24e6  120e6
    Codec                                   version      args
    C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
    done... (4*X*1) iteration(s)).
    m% ./fsbench -b4096 fast miniz,1 zlib,1 /mnt/usr/home/m/bench/scc1.tar
    Codec                                   version      args
    C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
    lrrle                                   0            256
       12281616 (x 1.026)     2776 MB/s 4793 MB/s        69e6  119e6
    LZ4                                     r114         
        8400786 (x 1.499)      316 MB/s 1180 MB/s       105e6  392e6
    LZF                                     3.6          very
        8072630 (x 1.560)      149 MB/s  482 MB/s        53e6  170e6
    LZO                                     2.06         1x1
        8256886 (x 1.525)      349 MB/s  622 MB/s       120e6  212e6
    QuickLZ                                 1.5.1b6      1
        8278188 (x 1.521)      240 MB/s  299 MB/s        82e6   95e6
    Snappy                                  1.1.0        
        8242726 (x 1.528)      330 MB/s  786 MB/s       113e6  265e6
    ZSTD                                    0.0.1        
        7343311 (x 1.715)     66.0 MB/s  234 MB/s        27e6   89e6
    miniz                                   1.11         1
        6757868 (x 1.864)     62.5 MB/s  123 MB/s        28e6   48e6
    zlib                                    1.2.8        1
        6445695 (x 1.954)     32.5 MB/s  145 MB/s        15e6   62e6
    Codec                                   version      args
    C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
    done... (4*X*1) iteration(s)).
    The Decoding failed issues are probably in fsbench, don't waste your time on them yet.

  22. #18
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Fixed the issue, it was in zstd integration. Also, I see there's code that would enable context preservation in zstd.c that isn't exposed to zstd.h. Would it noticeably improve small block efficiency if you exposed it to the user, possibly with some re-init function?

  23. #19
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Code:
    m% please ./fsbench -b131072 -m4096 fast miniz,1 zlib,1 /mnt/usr/home/m/bench/scc1.tar
    Codec                                   version      args
    C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
    lrrle                                   0            256
       12328960 (x 1.022)     3009 MB/s 5370 MB/s        63e6  113e6
    LZ4                                     r114        
        7503872 (x 1.678)      328 MB/s 1129 MB/s       132e6  456e6
    LZF                                     3.6          very
        7303168 (x 1.725)      169 MB/s  503 MB/s        71e6  208e6
    LZO                                     2.06         1x1
        7372800 (x 1.708)      380 MB/s  536 MB/s       157e6  217e6
    QuickLZ                                 1.5.1b6      1
        7073792 (x 1.781)      343 MB/s  330 MB/s       150e6  142e6
    Shrinker                                r6          
        7024640 (x 1.793)      218 MB/s  681 MB/s        96e6  296e6
    Snappy                                  1.1.0       
        7266304 (x 1.733)      320 MB/s  726 MB/s       135e6  303e6
    ZSTD                                    0.0.1       
        6078464 (x 2.072)      152 MB/s  407 MB/s        78e6  210e6
    miniz                                   1.11         1
        6447104 (x 1.954)      102 MB/s  180 MB/s        49e6   78e6
    zlib                                    1.2.8        1
        5832704 (x 2.159)     42.7 MB/s  212 MB/s        22e6  104e6
    Codec                                   version      args
    C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
    done... (4*X*1) iteration(s)).
    Code:
    m% please ./fsbench -b4096 -m512 fast miniz,1 zlib,1 /mnt/usr/home/m/bench/scc1.tar
    Codec                                   version      args
    C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
    lrrle                                   0            256
       12382208 (x 1.017)     2765 MB/s 4881 MB/s        46e6   82e6
    LZ4                                     r114        
        9083904 (x 1.387)      320 MB/s 1111 MB/s        89e6  306e6
    LZF                                     3.6          very
        8790016 (x 1.433)      154 MB/s  479 MB/s        46e6  144e6
    LZO                                     2.06         1x1
        8924160 (x 1.411)      349 MB/s  566 MB/s       101e6  160e6
    QuickLZ                                 1.5.1b6      1
        8901632 (x 1.415)      224 MB/s  316 MB/s        65e6   87e6
    Shrinker                                r6          
        8773632 (x 1.436)      183 MB/s  674 MB/s        55e6  201e6
    Snappy                                  1.1.0       
        8878080 (x 1.419)      329 MB/s  725 MB/s        97e6  208e6
    ZSTD                                    0.0.1       
        8370688 (x 1.505)     78.6 MB/s  245 MB/s        26e6   77e6
    miniz                                   1.11         1
        7370752 (x 1.709)     62.7 MB/s  120 MB/s        25e6   44e6
    zlib                                    1.2.8        1
        7216128 (x 1.745)     32.5 MB/s  141 MB/s        13e6   55e6
    Codec                                   version      args
    C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
    done... (4*X*1) iteration(s)).
    Code:
    m% please ./fsbench fast miniz,1 zlib,1 /mnt/usr/home/m/bench/HANNOMB.ttf
    Codec                                   version      args
    C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
    lrrle                                   0            256
       28519328 (x 1.186)     3212 MB/s 3250 MB/s       503e6  499e6
    LZ4                                     r114        
       22643128 (x 1.493)      377 MB/s 1562 MB/s       124e6  516e6
    LZF                                     3.6          very
       22553772 (x 1.499)      163 MB/s  555 MB/s        54e6  184e6
    LZO                                     2.06         1x1
       22828942 (x 1.481)      473 MB/s  797 MB/s       153e6  258e6
    QuickLZ                                 1.5.1b6      1
       23968479 (x 1.411)      295 MB/s  290 MB/s        85e6   84e6
    Shrinker                                r6          
       21749522 (x 1.555)      232 MB/s  974 MB/s        82e6  347e6
    Snappy                                  1.1.0       
       22766354 (x 1.485)      416 MB/s 1077 MB/s       135e6  351e6
    ZSTD                                    0.0.1       
       18808770 (x 1.798)      117 MB/s  378 MB/s        52e6  167e6
    miniz                                   1.11         1
       20861729 (x 1.621)     95.7 MB/s  199 MB/s        36e6   76e6
    zlib                                    1.2.8        1
       19608083 (x 1.725)     33.0 MB/s  190 MB/s        13e6   79e6
    Codec                                   version      args
    C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
    done... (4*X*1) iteration(s)).
    Code:
    m% please ./fsbench -b131072 -m4096 fast miniz,1 zlib,1 /mnt/usr/home/m/bench/HANNOMB.ttf
    Codec                                   version      args
    C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
    lrrle                                   0            256
       28622848 (x 1.181)     3174 MB/s 4668 MB/s       487e6  713e6
    LZ4                                     r114        
       23244800 (x 1.455)      380 MB/s 1542 MB/s       118e6  463e6
    LZF                                     3.6          very
       23166976 (x 1.460)      159 MB/s  554 MB/s        49e6  155e6
    LZO                                     2.06         1x1
       23347200 (x 1.448)      475 MB/s  809 MB/s       147e6  232e6
    QuickLZ                                 1.5.1b6      1
       24723456 (x 1.368)      289 MB/s  287 MB/s        77e6   62e6
    Shrinker                                r6          
       22446080 (x 1.507)      227 MB/s  943 MB/s        76e6  297e6
    Snappy                                  1.1.0       
       23306240 (x 1.451)      413 MB/s 1067 MB/s       128e6  313e6
    ZSTD                                    0.0.1       
       19718144 (x 1.715)      121 MB/s  370 MB/s        50e6  129e6
    miniz                                   1.11         1
       21422080 (x 1.579)     95.1 MB/s  199 MB/s        34e6   72e6
    zlib                                    1.2.8        1
       20242432 (x 1.671)     35.6 MB/s  187 MB/s        14e6   75e6
    Codec                                   version      args
    C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
    done... (4*X*1) iteration(s)).
    Code:
    m% please ./fsbench -b4096 -m512 fast miniz,1 zlib,1 /mnt/usr/home/m/bench/HANNOMB.ttf  
    Codec                                   version      args
    C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
    lrrle                                   0            256
       29169152 (x 1.159)     2891 MB/s 4161 MB/s       397e6  567e6
    LZ4                                     r114        
       26068992 (x 1.297)      398 MB/s 1499 MB/s        91e6  335e6
    LZF                                     3.6          very
       25794048 (x 1.311)      135 MB/s  574 MB/s        32e6  121e6
    LZO                                     2.06         1x1
       26704384 (x 1.266)      487 MB/s  933 MB/s       102e6  195e6
    QuickLZ                                 1.5.1b6      1
       28376576 (x 1.192)      224 MB/s  478 MB/s        36e6   71e6
    Shrinker                                r6          
       25590272 (x 1.321)      211 MB/s  818 MB/s        51e6  197e6
    Snappy                                  1.1.0       
       26014720 (x 1.300)      439 MB/s 1136 MB/s       101e6  249e6
    ZSTD                                    0.0.1       
       28322304 (x 1.194)     98.5 MB/s  432 MB/s        15e6   60e6
    miniz                                   1.11         1
       24563712 (x 1.377)     61.2 MB/s  135 MB/s        16e6   36e6
    zlib                                    1.2.8        1
       23489024 (x 1.440)     30.5 MB/s  137 MB/s      9549e3   41e6
    Codec                                   version      args
    C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
    done... (4*X*1) iteration(s)).
    Code:
    m% please ./fsbench fast miniz,1 zlib,1 /mnt/usr/home/m/bench/TCUP.tar                  
    Codec                                   version      args
    C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
    lrrle                                   0            256
      248183056 (x 1.052)     3057 MB/s 3117 MB/s       150e6  131e6
    LZ4                                     r114        
      159603695 (x 1.635)      364 MB/s 1310 MB/s       141e6  509e6
    LZF                                     3.6          very
      160578055 (x 1.626)      165 MB/s  526 MB/s        63e6  202e6
    LZO                                     2.06         1x1
      159872242 (x 1.633)      396 MB/s  622 MB/s       153e6  240e6
    QuickLZ                                 1.5.1b6      1
      158215112 (x 1.650)      311 MB/s  306 MB/s       122e6  120e6
    Shrinker                                r6          
      149601338 (x 1.745)      221 MB/s  764 MB/s        94e6  325e6
    Snappy                                  1.1.0       
      160344933 (x 1.628)      363 MB/s  842 MB/s       139e6  324e6
    ZSTD                                    0.0.1       
      135362108 (x 1.928)      144 MB/s  453 MB/s        69e6  218e6
    miniz                                   1.11         1
      143305902 (x 1.821)      100 MB/s  189 MB/s        45e6   85e6
    zlib                                    1.2.8        1
      134781348 (x 1.937)     36.2 MB/s  215 MB/s        17e6  103e6
    Codec                                   version      args
    C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
    done... (4*X*1) iteration(s)).
    Code:
    m% please ./fsbench -b131072 -m4096 fast miniz,1 zlib,1 /mnt/usr/home/m/bench/TCUP.tar   
    Codec                                   version      args
    C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
    lrrle                                   0            256
      250183680 (x 1.043)     3063 MB/s 3208 MB/s       127e6  119e6
    LZ4                                     r114         
      164339712 (x 1.588)      369 MB/s 1231 MB/s       136e6  340e6
    LZF                                     3.6          very
      164024320 (x 1.591)      161 MB/s  485 MB/s        59e6  101e6
    LZO                                     2.06         1x1
      163831808 (x 1.593)      374 MB/s  566 MB/s       139e6  118e6
    QuickLZ                                 1.5.1b6      1
      160919552 (x 1.622)      330 MB/s  319 MB/s       126e6    0e0
    Shrinker                                r6           
      155152384 (x 1.682)      215 MB/s  704 MB/s        87e6  159e6
    Snappy                                  1.1.0        
      164102144 (x 1.591)      360 MB/s  775 MB/s       133e6  161e6
    ZSTD                                    0.0.1        
      145240064 (x 1.797)      152 MB/s  400 MB/s        67e6    0e0
    miniz                                   1.11         1
      147427328 (x 1.771)     99.1 MB/s  185 MB/s        43e6    0e0
    zlib                                    1.2.8        1
      139239424 (x 1.875)     38.7 MB/s  199 MB/s        18e6    0e0
    Codec                                   version      args
    C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
    done... (4*X*1) iteration(s)).
    Code:
    m% please ./fsbench -b4096 -m512 fast miniz,1 zlib,1 /mnt/usr/home/m/bench/TCUP.tar      
    Codec                                   version      args
    C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
    lrrle                                   0            256
      251005952 (x 1.040)     2771 MB/s 2974 MB/s       106e6   95e6
    LZ4                                     r114         
      190118912 (x 1.373)      350 MB/s 1150 MB/s        94e6  262e6
    LZF                                     3.6          very
      187048960 (x 1.396)      147 MB/s  467 MB/s        41e6   82e6
    LZO                                     2.06         1x1
      187856896 (x 1.390)      358 MB/s  607 MB/s       100e6  110e6
    QuickLZ                                 1.5.1b6      1
      187127808 (x 1.395)      222 MB/s  299 MB/s        62e6    0e0
    Shrinker                                r6           
      183848960 (x 1.420)      193 MB/s  688 MB/s        57e6  126e6
    Snappy                                  1.1.0        
      188742656 (x 1.383)      355 MB/s  777 MB/s        98e6  135e6
    ZSTD                                    0.0.1        
      184564736 (x 1.414)     67.8 MB/s  222 MB/s        19e6    0e0
    miniz                                   1.11         1
      164529152 (x 1.587)     58.7 MB/s  127 MB/s        21e6    0e0
    zlib                                    1.2.8        1
      160199168 (x 1.629)     31.7 MB/s  137 MB/s        12e6    0e0
    Codec                                   version      args
    C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
    done... (4*X*1) iteration(s)).
    Code:
    m% please ./fsbench fast miniz,1 zlib,1 /mnt/usr/home/m/bench/random.tar              
    Codec                                   version      args
    C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
    lrrle                                   0            256
      114239542 (x 1.001)     3010 MB/s 2924 MB/s      2015e3 1817e3
    LZ4                                     r114         
      114258533 (x 1.000)     1890 MB/s 2779 MB/s       944e3 1387e3
    LZF                                     3.6          very
      114314278 (x 1.000)      138 MB/s         -         0e0    0e0
    LZO                                     2.06         1x1
      114195262 (x 1.001)     1989 MB/s 2801 MB/s      2120e3 2986e3
    QuickLZ                                 1.5.1b6      1
      114314278 (x 1.000)      410 MB/s         -         0e0    0e0
    Shrinker                                r6           
      113839432 (x 1.004)      218 MB/s 2649 MB/s       927e3   11e6
    Snappy                                  1.1.0        
      113696628 (x 1.005)     1677 MB/s 2853 MB/s      9279e3   15e6
    ZSTD                                    0.0.1        
      113405393 (x 1.008)      302 MB/s 2848 MB/s      2462e3   22e6
    miniz                                   1.11         1
      113387149 (x 1.008)     78.4 MB/s  236 MB/s       651e3 1959e3
    zlib                                    1.2.8        1
      113372278 (x 1.008)     24.2 MB/s  401 MB/s       204e3 3382e3
    Codec                                   version      args
    C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
    done... (4*X*1) iteration(s)).
    Code:
    m% please ./fsbench fast miniz,1 zlib,1 /mnt/usr/home/m/bench/calgary.tar 
    Codec                                   version      args
    C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
    lrrle                                   0            256
        2993824 (x 1.053)     4546 MB/s 4811 MB/s       229e6  242e6
    LZ4                                     r114         
        1620566 (x 1.946)      289 MB/s 1078 MB/s       140e6  523e6
    LZF                                     3.6          very
        1534640 (x 2.054)      174 MB/s  441 MB/s        89e6  226e6
    LZO                                     2.06         1x1
        1595374 (x 1.976)      323 MB/s  447 MB/s       159e6  221e6
    QuickLZ                                 1.5.1b6      1
        1497111 (x 2.106)      354 MB/s  333 MB/s       185e6  174e6
    Shrinker                                r6           
        1482747 (x 2.126)      213 MB/s  600 MB/s       112e6  317e6
    Snappy                                  1.1.0        
        1619277 (x 1.947)      289 MB/s  624 MB/s       140e6  303e6
    ZSTD                                    0.0.1        
        1121586 (x 2.811)      131 MB/s  371 MB/s        84e6  238e6
    miniz                                   1.11         1
        1372044 (x 2.298)      103 MB/s  162 MB/s        58e6   91e6
    zlib                                    1.2.8        1
        1205442 (x 2.616)     43.5 MB/s  217 MB/s        26e6  134e6
    Codec                                   version      args
    C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
    done... (4*X*1) iteration(s)).
    Code:
    m% please ./fsbench fast miniz,1 zlib,1 /mnt/usr/home/m/bench/nbbs.tar 
    Codec                                   version      args
    C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
    lrrle                                   0            256
       26235392 (x 1.155)     3152 MB/s 3216 MB/s       421e6  422e6
    LZ4                                     r114         
       11392989 (x 2.659)      540 MB/s 1589 MB/s       336e6  991e6
    LZF                                     3.6          very
       12578199 (x 2.408)      221 MB/s  659 MB/s       129e6  385e6
    LZO                                     2.06         1x1
       12080864 (x 2.507)      568 MB/s  752 MB/s       341e6  451e6
    QuickLZ                                 1.5.1b6      1
       10901377 (x 2.778)      423 MB/s  473 MB/s       270e6  302e6
    Shrinker                                r6           
       10664293 (x 2.840)      315 MB/s  980 MB/s       204e6  635e6
    Snappy                                  1.1.0        
       12002468 (x 2.524)      519 MB/s 1053 MB/s       313e6  635e6
    ZSTD                                    0.0.1        
        5981341 (x 5.064)      334 MB/s  946 MB/s       267e6  759e6
    miniz                                   1.11         1
       11098920 (x 2.729)      144 MB/s  249 MB/s        91e6  158e6
    zlib                                    1.2.8        1
       10258576 (x 2.953)     53.4 MB/s  321 MB/s        35e6  212e6
    Codec                                   version      args
    C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
    done... (4*X*1) iteration(s)).
    Code:
    m% please ./fsbench fast miniz,1 zlib,1 /mnt/usr/home/m/bench/book1 
    Codec                                   version      args
    C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
    lrrle                                   0            256
         768771 (x 1.000)     5041 MB/s         -         0e0    0e0
    LZ4                                     r114         
         514574 (x 1.494)      213 MB/s  859 MB/s        70e6  284e6
    LZF                                     3.6          very
         465804 (x 1.650)      143 MB/s  383 MB/s        56e6  150e6
    LZO                                     2.06         1x1
         495312 (x 1.552)      242 MB/s  340 MB/s        86e6  120e6
    QuickLZ                                 1.5.1b6      1
         468025 (x 1.643)      318 MB/s  274 MB/s       124e6  107e6
    Shrinker                                r6           
         469750 (x 1.637)      171 MB/s  493 MB/s        66e6  191e6
    Snappy                                  1.1.0        
         508506 (x 1.512)      215 MB/s  464 MB/s        72e6  157e6
    ZSTD                                    0.0.1        
         335806 (x 2.289)      101 MB/s  299 MB/s        56e6  168e66
    miniz                                   1.11         1
         421496 (x 1.824)     79.4 MB/s  127 MB/s        35e6   57e6
    zlib                                    1.2.8        1
         367790 (x 2.090)     36.5 MB/s  183 MB/s        19e6   95e6
    Codec                                   version      args
    C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
    done... (4*X*1) iteration(s)).
    On small blocks miniz is clearly better and Shrinker is often too.
    Performance on nbbs....just wow.

    ADDED:
    Code:
    m% please ./fsbench -b65536 fast miniz,1 zlib,1 /mnt/usr/home/m/bench/nbbs.tar      
    Codec                                   version      args
    C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
    lrrle                                   0            256
       26235632 (x 1.155)     3146 MB/s 3227 MB/s       421e6  429e6
    LZ4                                     r114         
       12057544 (x 2.512)      563 MB/s 1499 MB/s       338e6  879e6
    LZF                                     3.6          very
       12664450 (x 2.392)      212 MB/s  629 MB/s       123e6  360e6
    LZO                                     2.06         1x1
       12382086 (x 2.446)      540 MB/s  705 MB/s       319e6  415e6
    QuickLZ                                 1.5.1b6      1
       11989518 (x 2.526)      414 MB/s  480 MB/s       250e6  261e6
    Shrinker                                r6           
       11488986 (x 2.636)      292 MB/s  926 MB/s       181e6  568e6
    Snappy                                  1.1.0        
       12003717 (x 2.523)      507 MB/s 1013 MB/s       305e6  608e6
    ZSTD                                    0.0.1        
       10269432 (x 2.949)      207 MB/s  553 MB/s       137e6  355e6
    miniz                                   1.11         1
       11260776 (x 2.690)      136 MB/s  243 MB/s        85e6  123e6
    zlib                                    1.2.8        1
       10512734 (x 2.881)     62.5 MB/s  298 MB/s        40e6  164e6
    Codec                                   version      args
    C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
    done... (4*X*1) iteration(s)).
    Size matters. Larger window helps a lot, but even w/out it zstd is great on this file....I'd take Shrinker though.
    Last edited by m^2; 29th January 2015 at 23:08.

  24. #20
    Member
    Join Date
    Aug 2013
    Location
    France
    Posts
    77
    Thanks
    27
    Thanked 26 Times in 11 Posts
    @m^2

    Since you don't want to use git for philosophical reasons, I created a mirror of your fossil project on github (https://github.com/gpnuma/fsbench) so everybody can use it to create their codec's forks. BTW I could not see the ZSTD codebase included so I created a submodule for it.
    For density integration, I created this repository : https://github.com/centaurean/fsbench-density which is a fork of the previous project.

    Here are the results for enwik8 :

    Codec version args
    C.Size (C.Ratio) E.Speed D.Speed E.Eff. D.Eff.
    density::chameleon 0.11.2
    61524474 (x 1.625) 811 MB/s 1111 MB/s 312e6 427e6
    density::mandala 0.11.2
    53156746 (x 1.881) 455 MB/s 427 MB/s 213e6 200e6
    LZ4 r114
    56973116 (x 1.755) 275 MB/s 1113 MB/s 118e6 478e6
    LZO 2.06 1x1
    55792795 (x 1.792) 287 MB/s 372 MB/s 126e6 164e6
    QuickLZ 1.5.1b6 1
    52334371 (x 1.911) 281 MB/s 347 MB/s 134e6 165e6
    Snappy 1.1.0
    56539845 (x 1.769) 239 MB/s 774 MB/s 103e6 336e6
    ZSTD 0.0.1
    39573323 (x 2.527) 172 MB/s 377 MB/s 103e6 227e6
    Codec version args
    C.Size (C.Ratio) E.Speed D.Speed E.Eff. D.Eff.
    done... (4*X*1) iteration(s)).

    @Cyan

    It would be really nice if you could integrate those results (or results you make up on your platform) to the ZSTD benchmark numbers. Thanks !

  25. Thanks (2):

    Bulat Ziganshin (29th January 2015),jibz (30th January 2015)

  26. #21
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    gpnuma, is it possible to download a patch from github? I can only browse it....
    Thx for a hint on missing zstd code, fixed now.

  27. #22
    Member
    Join Date
    Aug 2013
    Location
    France
    Posts
    77
    Thanks
    27
    Thanked 26 Times in 11 Posts
    Quote Originally Posted by m^2 View Post
    gpnuma, is it possible to download a patch from github? I can only browse it....
    Thx for a hint on missing zstd code, fixed now.
    Sure you just need to click on the commit you want, and append a .patch to the url on github.
    For example on github the density addon to fsbench is :
    https://github.com/centaurean/fsbenc...133fdbba369ada

    To to get a patch for this, just point your browser to the following url :
    https://github.com/centaurean/fsbenc...ba369ada.patch

  28. #23
    Member
    Join Date
    Sep 2008
    Location
    France
    Posts
    870
    Thanks
    471
    Thanked 264 Times in 109 Posts
    Quote Originally Posted by m^2 View Post
    I see there's code that would enable context preservation in zstd.c that isn't exposed to zstd.h. Would it noticeably improve small block efficiency if you exposed it to the user, possibly with some re-init function?
    Yes, definitely.
    Zstd is not yet friendly to small data blocks.
    But the basic mechanics are there though,
    so now it's mostly a matter of properly polishing the interface and expose it.



    Quote Originally Posted by gpnuma View Post
    @Cyan
    It would be really nice if you could integrate those results (or results you make up on your platform) to the ZSTD benchmark numbers. Thanks !
    Sure !
    Last edited by Cyan; 31st January 2015 at 14:34.

  29. #24
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts

  30. #25
    Member
    Join Date
    Sep 2008
    Location
    France
    Posts
    870
    Thanks
    471
    Thanked 264 Times in 109 Posts
    Thanks for notification

    I've tested the file linked in this thread, it seems to pass tests properly.

    It could be that one of the recent changes fixed the issue.
    Could you try updating the your zstd version from github ?
    I just merged the "dev" branch into "master".

    As a sidenote, the file downloaded seems to be different than the one benchmarked,
    since its size is 442368 bytes, while the table suggests the benchmarked file should be 3471360 bytes.

  31. #26
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Oh dear, something's wrong with the file indeed.

  32. #27
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Updates to ZSTD have fixed the issue.

    But there's another: with some compilers you #include fse, with some others you don't.
    It means that a makefile has to be compiler-aware. Can't you just #include it always?

  33. Thanks:

    avitar (2nd February 2015)

  34. #28
    Member
    Join Date
    Aug 2014
    Location
    Argentina
    Posts
    536
    Thanks
    236
    Thanked 90 Times in 70 Posts
    Code:
    *** stack smashing detected ***: zstd terminated
    Linux binary. X86.
    Reading from stdin. Writing to disk.

  35. #29
    Member
    Join Date
    Sep 2008
    Location
    France
    Posts
    870
    Thanks
    471
    Thanked 264 Times in 109 Posts
    @Gonzalo : A second fix is on the way, which I feel is related to your issue. Should be published soon within "dev" branch.

    @m^2 : Good point. I've been wondering about this point for some time.
    I'm even wondering if it wouldn't be better to completely integrate "fse.c", as in "copy / paste", within zstd.c
    It would reduce both the problems regarding #include and number of files a client project needs to include.

    The downside is that zstd.c would become very large.

  36. #30
    Member
    Join Date
    Jan 2015
    Location
    chennai,Tamilnadu,India
    Posts
    19
    Thanks
    2
    Thanked 0 Times in 0 Posts
    Zstd code is very good now, I want to have zstd in such a way that it takes input from an array instead of file or stdin. This is very useful for my work of integrating Zstd compression algorithm into c-store, a column oriented database. c-store gives columns to compression algorithm in form of char*. At present, it has LZW as compression technique and I want to replace it with Zstd. So, please release a version of zstd with input being taken from char* type and output being stored in another char* variable.

Page 1 of 14 12311 ... LastLast

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •