Zstandard

[Cyan]

Just for information :

I've recently released a first version of Zstd,
the open source successor of zhuff.

It's not finished yet, but this version seems in good enough shape for release and public comments.
You can download its source code on github.


While the decoding speed seems mostly settled,
the compression side, on the other hand, still requires a lot of work.
Only the "fast mode" is released so far.

An "HC" version can be created. It will require quite some time and expertise to reach its conclusion.

Of course, all comments and contributions are very welcomed.

[Bulat Ziganshin]

1. it's the most awkward name i've seen in compression
2. without repdist/repboth codes it will have less max. compression ratio than rar/tornado
3. Core i5-4300U has frequency up to 2.90 GHz, and i guess that it was actual freq in the test. you can use sensors window of hwinfo to check actual frequency during (de)compression
4. can you release the binary so i can check it here?

[Cyan]

Zstd includes a repDist code.
But the current version only takes passively advantage of it, rather than actively looking for it.
It's intended to be more useful at stronger compression modes (which still have to be developed).

Please find in attached file a binary version compiled using Visual 2012.

[m^2]

Yann, I see that you use fsbench. I'd love to get even a rough patch, so I can incorporate it in mine version faster.

[Bulat Ziganshin]

1. yes, you should set the compression format first, and then more backends will arrive. but in order to select the best compression format, you need to have several backends - from lz4-like one to optimum-parsing in order to check each decision in various compression domains. i may look into porting OP backend so that you can see the compression limits of the format

2. tornado includes 4 repdist codes (like rar/lzma), repboth code (len+dist) and funny "rep_dist1 + len:=1" code. also it includes table2/table4 codes that just diffs sequential 2/4-byte words prior to compression (tor.exe -t0 option disables table preprocessing, so you can check its effect). if you think that these features may slow down the decoder, you may provide multiple decoder levels - fastest one without (extra) rep codes. afair, repdist2..4 codes improved compression by 1..2%

3. tornado greedy/lazy compressor also has only passive repdist recognition, tornado OP as well as both lzma modes has active repdist recognition

[Matt Mahoney]

I updated LTCB and Silesia. On LTCB it makes the Pareto frontier for compression speed. It didn't for decompression probably because crush was run on a faster machine.
http://mattmahoney.net/dc/text.html#3546
http://mattmahoney.net/dc/silesia.html

Edit: for some reason, enwik9.zst was 1 byte bigger in the Windows test. (Decompression was OK in both tests).

[Gonzalo]

Frankly, I am impressed. Very good job for this very first version. I was looking for a good replacement for slugx, and finally I think I've found one.

Just informing you, compiling and compressing fine on Linux x86

BTW... I think most readers will understand your program's name as the 'Ultimate... well, STD'

[Gonzalo]

Error:

Read : 700 MB ==> 50.22% *** stack smashing detected ***: zstd terminated

Command: wine shar as - WINDOWS | wine fazip rep:400m - - | time zstd - WINDOWS.compress.

Are you trying to get root on my machine???

[Gonzalo]

Another one:

time wine slugx d WINDOWS.pack : | zstd - WINDOWS.comp

-------------------------------------------------------------------------------
 Slug X (Apr  6 2009) - Copyright (c) 2009 by Christian Martelock
-------------------------------------------------------------------------------

 Decompressing...
 src:    'WINDOWS.pack'
 dst:    ':'

Read : 1446 MB  ==> 52.21%   Error 24 : Compression error : ZSTD_ERROR_GENERIC 
 1010001.91 KB -> -0.00 KB (-0.00%, -1 bytes)

real    4m32.717s
user    3m12.644s
sys    0m14.488s

Apparently, both are related to memory pipeline. Perhaps is wine's fault. File-to-file jobs are correct. Also, "zstd -d in | zstd - out" performs correctly as well.

[gpnuma]

Very nice work !!

Here is an updated version of the efficiency benchmark I did a while ago, now including zstd .

Benchmark
Performed on a Macbook Pro, OSX 10.10.1, 2.3 GHz Intel Core i7, 8Go 1600 MHz DDR, SSD
All programs were compiled with their standard options (standard makefile) using clang (note that I also tried with gcc-4.9 and it makes no difference whatsoever). They were run a minimum of 5 times and chronometered using the best "user" value from the "time" utility. They are of course all single-threaded otherwise this would make no sense.
The file used was the world famous enwik8 but any compressible data would do !

Compression

efficiency = (1/(compression_ratio ^ power))/compression_time



X-axis : power (higher powers means ratio is considered more important than speed).
Y-axis : percentage of total (higher is better)

zstd is the most efficient when one thinks the ratio is a power of 5 at least more important than speed.
Below that sharc/density is clearly the most efficient.

Decompression

efficiency = (1/(compression_ratio ^ power))/decompression_time



X-axis : power (higher powers means ratio is considered more important than speed).
Y-axis : percentage of total (higher is better)

lz4 is ahead on decompression efficiency, but it's not really a surprise as it was optimized for that purpose.
lz4 -1, lz4 -9 and sharc -c1 decompress at roughly the same speeds but lz4 -9 's ratio is much better hence the results.

Round trip

This is usually the most interesting as it shows the general algorithm efficiency.

efficiency = (1/(compression_ratio ^ power))/roundtrip_time



X-axis : power (higher powers means ratio is considered more important than speed).
Y-axis : percentage of total (higher is better)

sharc/density is ahead when one considers that ratio is less than a power of 4 more important than speed.
After that zstd is ahead because of a better ratio.

PS. it would be nice if you could add the sharc/density results in your benchmark on your zstd github repository Cyan .

[Cyan]

> Yann, I see that you use fsbench. I'd love to get even a rough patch, so I can incorporate it in mine version faster.

@m^2 : I had to make a lot of little changes to add zstd.
I don't know how I can retrieve them to provide them to you as a compatible patch with your version of fsbench.

Maybe as an easier intermediate step, I could send you the full source code of the current version I use ? (basically, v0.14.3 + zstd modifications)

On a longer term, I would like to suggest refactoring the code so that adding a codec could be achieved with some localized modifications (as opposed to spread into multiple files). But I do acknowledge it is probably a significant work.


> 1. yes, you should set the compression format first, and then more backends will arrive. but in order to select the best compression format, you need to have several backends - from lz4-like one to optimum-parsing in order to check each decision in various compression domains. i may look into porting OP backend so that you can see the compression limits of the format

@Bulat : You are totally right.
This is the current work ongoing.
It will be great thing if you can achieve such a port, I trust a lot your skills at high compression parsing.

As a sidenote, maybe I should modify the current compression code a bit, for a clearer separation between match-selection (parsing) and encoding.
This would allow to include multiple match/parsers reusing a common encoding code.
It's not that difficult, since it's almost already done. It probably just need a clearer/easier interface contract.


@matt : Thanks for testing !

> Edit: for some reason, enwik9.zst was 1 byte bigger in the Windows test. (Decompression was OK in both tests).

The 1 byte difference between Linux & Windows, even though harmless, is nonetheless unexpected.
I'll try to reproduce it to understand what's going on.

Sidenote : I intend to provide a Visual project directory, for people who prefer this IDE. Will it be useful to you ?


@Gonzalo :

> Apparently, both are related to memory pipeline. Perhaps is wine's fault. File-to-file jobs are correct. Also, "zstd -d in | zstd - out" performs correctly as well.

Thanks for reporting
So, my understanding is that you used the Visual compiled windows binary in combination with Wine.
It's indeed a combination that was never tested before.
Maybe you found a bug on the pipe implementation.
It's unclear if it can be solved from source code side. Worth investigating.

In any case, if you are using a Linux machine, maybe compiling with a native Linux compiler (gcc, clang, etc.) will help you complete your tests.


@gpnuma :
Thanks for the kind words.
Your findings are very instructive.

> PS. it would be nice if you could add the sharc/density results in your benchmark on your zstd github repository Cyan .

Sure. As you may know, I'm currently using m^2's fsbench for testings.
fsbench doesn't yet integrate density, which is the reason why it's not yet present in the comparison.
This can probably be improved, I may add it on my own, or even switch to a different test code able to produce the comparison exercise.

Note that I tend to prefer the Silesia Corpus as a reference.
Enwik is a famous reference, rightly so thanks to Matt,
but it also stress a well-defined workload (text files), which, alone, is not representative enough of the various inputs a codec is expected to deal with.

[m^2]

> Yann, I see that you use fsbench. I'd love to get even a rough patch, so I can incorporate it in mine version faster.

@m^2 : I had to make a lot of little changes to add zstd.
I don't know how I can retrieve them to provide them to you as a compatible patch with your version of fsbench.

Maybe as an easier intermediate step, I could send you the full source code of the current version I use ? (basically, v0.14.3 + zstd modifications)

On a longer term, I would like to suggest refactoring the code so that adding a codec could be achieved with some localized modifications (as opposed to spread into multiple files). But I do acknowledge it is probably a significant work.

There are indeed many steps involved in adding a new codec, though they are documented in codecs.hpp. Did you see it? I doced it for my own needs, but a README might be a better place.
Putting all changes in a single place is not something I tried to do. I will thing about it.

I'll gladly take the source.

@gpnuma :
Thanks for the kind words.
Your findings are very instructive.

> PS. it would be nice if you could add the sharc/density results in your benchmark on your zstd github repository Cyan .

Sure. As you may know, I'm currently using m^2's fsbench for testings.
fsbench doesn't yet integrate density, which is the reason why it's not yet present in the comparison.
This can probably be improved, I may add it on my own, or even switch to a different test code able to produce the comparison exercise.

Note that I tend to prefer the Silesia Corpus as a reference.
Enwik is a famous reference, rightly so thanks to Matt,
but it also stress a well-defined workload (text files), which, alone, is not representative enough of the various inputs a codec is expected to deal with.

I tried to add libssc to fsbench, harder than I would usually because of reported performance, but couldn't make it work, the code just seemed broken. I noticed that gpnuma made some updates recently and intend to try again, but I have neither much time nor much faith in success, so zstd is a higher priority task for me.

[Gonzalo]

So, my understanding is that you used the Visual compiled windows binary in combination with Wine.

Nope. I compiled myself Zstd using 'make install' as is the source and MAKEFILE in the package. That means I'm using a proper Linux ELF. There are also some windows programs throwing data to it via pipeline.
Windows programs are Shelwien's shar, Bulat's fazip and Christian's slugx, each one on different runs to make me sure.

[gpnuma]

harder than I would usually because of reported performance, but couldn't make it work, the code just seemed broken. I noticed that gpnuma made some updates recently and intend to try again, but I have neither much time nor much faith in success, so zstd is a higher priority task for me.

Lol you're right the master branch was broken at the time that's why I wanted you to checkout the latest beta release instead

IMO what you should do m^2 is publish your code on github (it's free) !
I will then create all the modifications necessary in a fork and send you a pull request , this would actually be the simplest for everyone to add a codec to your benchmark hence insuring its well deserved success !

[m^2]

Beta was broken the same.
I prefer chiselapp:
https://chiselapp.com/user/Justin_be...itory/fsbench/

it serves the same purpose and unlike git, fossil is free from copyleft.

[Cyan]

@m^2 : Please find in attached file
the modified source directory of fsbench 0.14.3
including the modifications to integrate Zstd.

Hope it can help

[m^2]

ZSTD support is added it fsbench.
Clang 3.3 -O3, Phenom 2@3.2 ghz:

m% ./fsbench fast miniz,1 zlib,1 /mnt/usr/home/m/bench/scc1.tar
Codec                                   version      args
C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
lrrle                                   0            256
   12281040 (x 1.026)     3069 MB/s 3087 MB/s        76e6   76e6
LZ4                                     r114         
    7201745 (x 1.749)      328 MB/s 1139 MB/s       140e6  487e6
LZF                                     3.6          very
    7090337 (x 1.776)      176 MB/s  507 MB/s        76e6  221e6
LZO                                     2.06         1x1
    7152460 (x 1.761)      383 MB/s  601 MB/s       165e6  259e6
QuickLZ                                 1.5.1b6      1
    6804291 (x 1.851)      336 MB/s  327 MB/s       154e6  150e6
Snappy                                  1.1.0        
    7070230 (x 1.781)      322 MB/s  745 MB/s       141e6  326e6
ZSTD                                    0.0.1        
    5510621 (x 2.286)      141 MB/s  405 MB/s        79e6  227e6
miniz                                   1.11         1
    6245122 (x 2.017)      102 MB/s  180 MB/s        51e6   90e6
zlib                                    1.2.8        1
    5583512 (x 2.256)     40.9 MB/s  223 MB/s        22e6  124e6
Codec                                   version      args
C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
done... (4*X*1) iteration(s)).

m% ./fsbench -b131072 -m4096 fast miniz,1 zlib,1 /mnt/usr/home/m/bench/scc1.tar
Codec                                   version      args
C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
lrrle                                   0            256
   12328960 (x 1.022)     3009 MB/s 5317 MB/s        63e6  112e6
LZ4                                     r114         
    7503872 (x 1.678)      325 MB/s 1130 MB/s       131e6  447e6
LZF                                     3.6          very
    7303168 (x 1.725)      170 MB/s  496 MB/s        71e6  205e6
LZO                                     2.06         1x1
    7372800 (x 1.708)      378 MB/s  537 MB/s       156e6  217e6
QuickLZ                                 1.5.1b6      1
    7073792 (x 1.781)      339 MB/s  330 MB/s       148e6  142e6
Snappy                                  1.1.0        
    7266304 (x 1.733)      321 MB/s  728 MB/s       135e6  304e6
ZSTD                                    0.0.1        
Decoding failed!%            
                m% ./fsbench -b4096 -m512 fast miniz,1 zlib,1 /mnt/usr/home/m/bench/scc1.tar
Codec                                   version      args
C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
lrrle                                   0            256
   12382208 (x 1.017)     2765 MB/s 4691 MB/s        46e6   79e6
LZ4                                     r114         
    9083904 (x 1.387)      320 MB/s 1111 MB/s        89e6  306e6
LZF                                     3.6          very
    8790016 (x 1.433)      154 MB/s  472 MB/s        46e6  142e6
LZO                                     2.06         1x1
    8924160 (x 1.411)      349 MB/s  567 MB/s       101e6  160e6
QuickLZ                                 1.5.1b6      1
    8901632 (x 1.415)      240 MB/s  319 MB/s        70e6   88e6
Snappy                                  1.1.0        
    8878080 (x 1.419)      329 MB/s  726 MB/s        97e6  209e6
ZSTD                                    0.0.1        
Decoding failed!%    
                     m% ./fsbench -b131072 fast miniz,1 zlib,1 /mnt/usr/home/m/bench/scc1.tar 
Codec                                   version      args
C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
lrrle                                   0            256
   12281040 (x 1.026)     3021 MB/s 3315 MB/s        75e6   82e6
LZ4                                     r114         
    7267513 (x 1.733)      326 MB/s 1144 MB/s       137e6  483e6
LZF                                     3.6          very
    7119193 (x 1.769)      169 MB/s  500 MB/s        73e6  211e6
LZO                                     2.06         1x1
    7175287 (x 1.755)      377 MB/s  557 MB/s       162e6  234e6
QuickLZ                                 1.5.1b6      1
    6864057 (x 1.835)      338 MB/s  331 MB/s       153e6  148e6
Snappy                                  1.1.0        
    7070516 (x 1.781)      320 MB/s  739 MB/s       140e6  324e6
ZSTD                                    0.0.1        
    5885924 (x 2.140)      151 MB/s  410 MB/s        80e6  213e6
miniz                                   1.11         1
    6254371 (x 2.014)      100 MB/s  180 MB/s        50e6   90e6
zlib                                    1.2.8        1
    5628481 (x 2.238)     43.5 MB/s  218 MB/s        24e6  120e6
Codec                                   version      args
C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
done... (4*X*1) iteration(s)).
m% ./fsbench -b4096 fast miniz,1 zlib,1 /mnt/usr/home/m/bench/scc1.tar
Codec                                   version      args
C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
lrrle                                   0            256
   12281616 (x 1.026)     2776 MB/s 4793 MB/s        69e6  119e6
LZ4                                     r114         
    8400786 (x 1.499)      316 MB/s 1180 MB/s       105e6  392e6
LZF                                     3.6          very
    8072630 (x 1.560)      149 MB/s  482 MB/s        53e6  170e6
LZO                                     2.06         1x1
    8256886 (x 1.525)      349 MB/s  622 MB/s       120e6  212e6
QuickLZ                                 1.5.1b6      1
    8278188 (x 1.521)      240 MB/s  299 MB/s        82e6   95e6
Snappy                                  1.1.0        
    8242726 (x 1.528)      330 MB/s  786 MB/s       113e6  265e6
ZSTD                                    0.0.1        
    7343311 (x 1.715)     66.0 MB/s  234 MB/s        27e6   89e6
miniz                                   1.11         1
    6757868 (x 1.864)     62.5 MB/s  123 MB/s        28e6   48e6
zlib                                    1.2.8        1
    6445695 (x 1.954)     32.5 MB/s  145 MB/s        15e6   62e6
Codec                                   version      args
C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
done... (4*X*1) iteration(s)).

The Decoding failed issues are probably in fsbench, don't waste your time on them yet.

[m^2]

Fixed the issue, it was in zstd integration. Also, I see there's code that would enable context preservation in zstd.c that isn't exposed to zstd.h. Would it noticeably improve small block efficiency if you exposed it to the user, possibly with some re-init function?

[m^2]

m% please ./fsbench -b131072 -m4096 fast miniz,1 zlib,1 /mnt/usr/home/m/bench/scc1.tar
Codec                                   version      args
C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
lrrle                                   0            256
   12328960 (x 1.022)     3009 MB/s 5370 MB/s        63e6  113e6
LZ4                                     r114        
    7503872 (x 1.678)      328 MB/s 1129 MB/s       132e6  456e6
LZF                                     3.6          very
    7303168 (x 1.725)      169 MB/s  503 MB/s        71e6  208e6
LZO                                     2.06         1x1
    7372800 (x 1.708)      380 MB/s  536 MB/s       157e6  217e6
QuickLZ                                 1.5.1b6      1
    7073792 (x 1.781)      343 MB/s  330 MB/s       150e6  142e6
Shrinker                                r6          
    7024640 (x 1.793)      218 MB/s  681 MB/s        96e6  296e6
Snappy                                  1.1.0       
    7266304 (x 1.733)      320 MB/s  726 MB/s       135e6  303e6
ZSTD                                    0.0.1       
    6078464 (x 2.072)      152 MB/s  407 MB/s        78e6  210e6
miniz                                   1.11         1
    6447104 (x 1.954)      102 MB/s  180 MB/s        49e6   78e6
zlib                                    1.2.8        1
    5832704 (x 2.159)     42.7 MB/s  212 MB/s        22e6  104e6
Codec                                   version      args
C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
done... (4*X*1) iteration(s)).

m% please ./fsbench -b4096 -m512 fast miniz,1 zlib,1 /mnt/usr/home/m/bench/scc1.tar
Codec                                   version      args
C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
lrrle                                   0            256
   12382208 (x 1.017)     2765 MB/s 4881 MB/s        46e6   82e6
LZ4                                     r114        
    9083904 (x 1.387)      320 MB/s 1111 MB/s        89e6  306e6
LZF                                     3.6          very
    8790016 (x 1.433)      154 MB/s  479 MB/s        46e6  144e6
LZO                                     2.06         1x1
    8924160 (x 1.411)      349 MB/s  566 MB/s       101e6  160e6
QuickLZ                                 1.5.1b6      1
    8901632 (x 1.415)      224 MB/s  316 MB/s        65e6   87e6
Shrinker                                r6          
    8773632 (x 1.436)      183 MB/s  674 MB/s        55e6  201e6
Snappy                                  1.1.0       
    8878080 (x 1.419)      329 MB/s  725 MB/s        97e6  208e6
ZSTD                                    0.0.1       
    8370688 (x 1.505)     78.6 MB/s  245 MB/s        26e6   77e6
miniz                                   1.11         1
    7370752 (x 1.709)     62.7 MB/s  120 MB/s        25e6   44e6
zlib                                    1.2.8        1
    7216128 (x 1.745)     32.5 MB/s  141 MB/s        13e6   55e6
Codec                                   version      args
C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
done... (4*X*1) iteration(s)).

m% please ./fsbench fast miniz,1 zlib,1 /mnt/usr/home/m/bench/HANNOMB.ttf
Codec                                   version      args
C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
lrrle                                   0            256
   28519328 (x 1.186)     3212 MB/s 3250 MB/s       503e6  499e6
LZ4                                     r114        
   22643128 (x 1.493)      377 MB/s 1562 MB/s       124e6  516e6
LZF                                     3.6          very
   22553772 (x 1.499)      163 MB/s  555 MB/s        54e6  184e6
LZO                                     2.06         1x1
   22828942 (x 1.481)      473 MB/s  797 MB/s       153e6  258e6
QuickLZ                                 1.5.1b6      1
   23968479 (x 1.411)      295 MB/s  290 MB/s        85e6   84e6
Shrinker                                r6          
   21749522 (x 1.555)      232 MB/s  974 MB/s        82e6  347e6
Snappy                                  1.1.0       
   22766354 (x 1.485)      416 MB/s 1077 MB/s       135e6  351e6
ZSTD                                    0.0.1       
   18808770 (x 1.798)      117 MB/s  378 MB/s        52e6  167e6
miniz                                   1.11         1
   20861729 (x 1.621)     95.7 MB/s  199 MB/s        36e6   76e6
zlib                                    1.2.8        1
   19608083 (x 1.725)     33.0 MB/s  190 MB/s        13e6   79e6
Codec                                   version      args
C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
done... (4*X*1) iteration(s)).

m% please ./fsbench -b131072 -m4096 fast miniz,1 zlib,1 /mnt/usr/home/m/bench/HANNOMB.ttf
Codec                                   version      args
C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
lrrle                                   0            256
   28622848 (x 1.181)     3174 MB/s 4668 MB/s       487e6  713e6
LZ4                                     r114        
   23244800 (x 1.455)      380 MB/s 1542 MB/s       118e6  463e6
LZF                                     3.6          very
   23166976 (x 1.460)      159 MB/s  554 MB/s        49e6  155e6
LZO                                     2.06         1x1
   23347200 (x 1.448)      475 MB/s  809 MB/s       147e6  232e6
QuickLZ                                 1.5.1b6      1
   24723456 (x 1.368)      289 MB/s  287 MB/s        77e6   62e6
Shrinker                                r6          
   22446080 (x 1.507)      227 MB/s  943 MB/s        76e6  297e6
Snappy                                  1.1.0       
   23306240 (x 1.451)      413 MB/s 1067 MB/s       128e6  313e6
ZSTD                                    0.0.1       
   19718144 (x 1.715)      121 MB/s  370 MB/s        50e6  129e6
miniz                                   1.11         1
   21422080 (x 1.579)     95.1 MB/s  199 MB/s        34e6   72e6
zlib                                    1.2.8        1
   20242432 (x 1.671)     35.6 MB/s  187 MB/s        14e6   75e6
Codec                                   version      args
C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
done... (4*X*1) iteration(s)).

m% please ./fsbench -b4096 -m512 fast miniz,1 zlib,1 /mnt/usr/home/m/bench/HANNOMB.ttf  
Codec                                   version      args
C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
lrrle                                   0            256
   29169152 (x 1.159)     2891 MB/s 4161 MB/s       397e6  567e6
LZ4                                     r114        
   26068992 (x 1.297)      398 MB/s 1499 MB/s        91e6  335e6
LZF                                     3.6          very
   25794048 (x 1.311)      135 MB/s  574 MB/s        32e6  121e6
LZO                                     2.06         1x1
   26704384 (x 1.266)      487 MB/s  933 MB/s       102e6  195e6
QuickLZ                                 1.5.1b6      1
   28376576 (x 1.192)      224 MB/s  478 MB/s        36e6   71e6
Shrinker                                r6          
   25590272 (x 1.321)      211 MB/s  818 MB/s        51e6  197e6
Snappy                                  1.1.0       
   26014720 (x 1.300)      439 MB/s 1136 MB/s       101e6  249e6
ZSTD                                    0.0.1       
   28322304 (x 1.194)     98.5 MB/s  432 MB/s        15e6   60e6
miniz                                   1.11         1
   24563712 (x 1.377)     61.2 MB/s  135 MB/s        16e6   36e6
zlib                                    1.2.8        1
   23489024 (x 1.440)     30.5 MB/s  137 MB/s      9549e3   41e6
Codec                                   version      args
C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
done... (4*X*1) iteration(s)).

m% please ./fsbench fast miniz,1 zlib,1 /mnt/usr/home/m/bench/TCUP.tar                  
Codec                                   version      args
C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
lrrle                                   0            256
  248183056 (x 1.052)     3057 MB/s 3117 MB/s       150e6  131e6
LZ4                                     r114        
  159603695 (x 1.635)      364 MB/s 1310 MB/s       141e6  509e6
LZF                                     3.6          very
  160578055 (x 1.626)      165 MB/s  526 MB/s        63e6  202e6
LZO                                     2.06         1x1
  159872242 (x 1.633)      396 MB/s  622 MB/s       153e6  240e6
QuickLZ                                 1.5.1b6      1
  158215112 (x 1.650)      311 MB/s  306 MB/s       122e6  120e6
Shrinker                                r6          
  149601338 (x 1.745)      221 MB/s  764 MB/s        94e6  325e6
Snappy                                  1.1.0       
  160344933 (x 1.628)      363 MB/s  842 MB/s       139e6  324e6
ZSTD                                    0.0.1       
  135362108 (x 1.928)      144 MB/s  453 MB/s        69e6  218e6
miniz                                   1.11         1
  143305902 (x 1.821)      100 MB/s  189 MB/s        45e6   85e6
zlib                                    1.2.8        1
  134781348 (x 1.937)     36.2 MB/s  215 MB/s        17e6  103e6
Codec                                   version      args
C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
done... (4*X*1) iteration(s)).

m% please ./fsbench -b131072 -m4096 fast miniz,1 zlib,1 /mnt/usr/home/m/bench/TCUP.tar   
Codec                                   version      args
C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
lrrle                                   0            256
  250183680 (x 1.043)     3063 MB/s 3208 MB/s       127e6  119e6
LZ4                                     r114         
  164339712 (x 1.588)      369 MB/s 1231 MB/s       136e6  340e6
LZF                                     3.6          very
  164024320 (x 1.591)      161 MB/s  485 MB/s        59e6  101e6
LZO                                     2.06         1x1
  163831808 (x 1.593)      374 MB/s  566 MB/s       139e6  118e6
QuickLZ                                 1.5.1b6      1
  160919552 (x 1.622)      330 MB/s  319 MB/s       126e6    0e0
Shrinker                                r6           
  155152384 (x 1.682)      215 MB/s  704 MB/s        87e6  159e6
Snappy                                  1.1.0        
  164102144 (x 1.591)      360 MB/s  775 MB/s       133e6  161e6
ZSTD                                    0.0.1        
  145240064 (x 1.797)      152 MB/s  400 MB/s        67e6    0e0
miniz                                   1.11         1
  147427328 (x 1.771)     99.1 MB/s  185 MB/s        43e6    0e0
zlib                                    1.2.8        1
  139239424 (x 1.875)     38.7 MB/s  199 MB/s        18e6    0e0
Codec                                   version      args
C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
done... (4*X*1) iteration(s)).

m% please ./fsbench -b4096 -m512 fast miniz,1 zlib,1 /mnt/usr/home/m/bench/TCUP.tar      
Codec                                   version      args
C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
lrrle                                   0            256
  251005952 (x 1.040)     2771 MB/s 2974 MB/s       106e6   95e6
LZ4                                     r114         
  190118912 (x 1.373)      350 MB/s 1150 MB/s        94e6  262e6
LZF                                     3.6          very
  187048960 (x 1.396)      147 MB/s  467 MB/s        41e6   82e6
LZO                                     2.06         1x1
  187856896 (x 1.390)      358 MB/s  607 MB/s       100e6  110e6
QuickLZ                                 1.5.1b6      1
  187127808 (x 1.395)      222 MB/s  299 MB/s        62e6    0e0
Shrinker                                r6           
  183848960 (x 1.420)      193 MB/s  688 MB/s        57e6  126e6
Snappy                                  1.1.0        
  188742656 (x 1.383)      355 MB/s  777 MB/s        98e6  135e6
ZSTD                                    0.0.1        
  184564736 (x 1.414)     67.8 MB/s  222 MB/s        19e6    0e0
miniz                                   1.11         1
  164529152 (x 1.587)     58.7 MB/s  127 MB/s        21e6    0e0
zlib                                    1.2.8        1
  160199168 (x 1.629)     31.7 MB/s  137 MB/s        12e6    0e0
Codec                                   version      args
C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
done... (4*X*1) iteration(s)).

m% please ./fsbench fast miniz,1 zlib,1 /mnt/usr/home/m/bench/random.tar              
Codec                                   version      args
C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
lrrle                                   0            256
  114239542 (x 1.001)     3010 MB/s 2924 MB/s      2015e3 1817e3
LZ4                                     r114         
  114258533 (x 1.000)     1890 MB/s 2779 MB/s       944e3 1387e3
LZF                                     3.6          very
  114314278 (x 1.000)      138 MB/s         -         0e0    0e0
LZO                                     2.06         1x1
  114195262 (x 1.001)     1989 MB/s 2801 MB/s      2120e3 2986e3
QuickLZ                                 1.5.1b6      1
  114314278 (x 1.000)      410 MB/s         -         0e0    0e0
Shrinker                                r6           
  113839432 (x 1.004)      218 MB/s 2649 MB/s       927e3   11e6
Snappy                                  1.1.0        
  113696628 (x 1.005)     1677 MB/s 2853 MB/s      9279e3   15e6
ZSTD                                    0.0.1        
  113405393 (x 1.008)      302 MB/s 2848 MB/s      2462e3   22e6
miniz                                   1.11         1
  113387149 (x 1.008)     78.4 MB/s  236 MB/s       651e3 1959e3
zlib                                    1.2.8        1
  113372278 (x 1.008)     24.2 MB/s  401 MB/s       204e3 3382e3
Codec                                   version      args
C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
done... (4*X*1) iteration(s)).

m% please ./fsbench fast miniz,1 zlib,1 /mnt/usr/home/m/bench/calgary.tar 
Codec                                   version      args
C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
lrrle                                   0            256
    2993824 (x 1.053)     4546 MB/s 4811 MB/s       229e6  242e6
LZ4                                     r114         
    1620566 (x 1.946)      289 MB/s 1078 MB/s       140e6  523e6
LZF                                     3.6          very
    1534640 (x 2.054)      174 MB/s  441 MB/s        89e6  226e6
LZO                                     2.06         1x1
    1595374 (x 1.976)      323 MB/s  447 MB/s       159e6  221e6
QuickLZ                                 1.5.1b6      1
    1497111 (x 2.106)      354 MB/s  333 MB/s       185e6  174e6
Shrinker                                r6           
    1482747 (x 2.126)      213 MB/s  600 MB/s       112e6  317e6
Snappy                                  1.1.0        
    1619277 (x 1.947)      289 MB/s  624 MB/s       140e6  303e6
ZSTD                                    0.0.1        
    1121586 (x 2.811)      131 MB/s  371 MB/s        84e6  238e6
miniz                                   1.11         1
    1372044 (x 2.298)      103 MB/s  162 MB/s        58e6   91e6
zlib                                    1.2.8        1
    1205442 (x 2.616)     43.5 MB/s  217 MB/s        26e6  134e6
Codec                                   version      args
C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
done... (4*X*1) iteration(s)).

m% please ./fsbench fast miniz,1 zlib,1 /mnt/usr/home/m/bench/nbbs.tar 
Codec                                   version      args
C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
lrrle                                   0            256
   26235392 (x 1.155)     3152 MB/s 3216 MB/s       421e6  422e6
LZ4                                     r114         
   11392989 (x 2.659)      540 MB/s 1589 MB/s       336e6  991e6
LZF                                     3.6          very
   12578199 (x 2.408)      221 MB/s  659 MB/s       129e6  385e6
LZO                                     2.06         1x1
   12080864 (x 2.507)      568 MB/s  752 MB/s       341e6  451e6
QuickLZ                                 1.5.1b6      1
   10901377 (x 2.778)      423 MB/s  473 MB/s       270e6  302e6
Shrinker                                r6           
   10664293 (x 2.840)      315 MB/s  980 MB/s       204e6  635e6
Snappy                                  1.1.0        
   12002468 (x 2.524)      519 MB/s 1053 MB/s       313e6  635e6
ZSTD                                    0.0.1        
    5981341 (x 5.064)      334 MB/s  946 MB/s       267e6  759e6
miniz                                   1.11         1
   11098920 (x 2.729)      144 MB/s  249 MB/s        91e6  158e6
zlib                                    1.2.8        1
   10258576 (x 2.953)     53.4 MB/s  321 MB/s        35e6  212e6
Codec                                   version      args
C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
done... (4*X*1) iteration(s)).

m% please ./fsbench fast miniz,1 zlib,1 /mnt/usr/home/m/bench/book1 
Codec                                   version      args
C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
lrrle                                   0            256
     768771 (x 1.000)     5041 MB/s         -         0e0    0e0
LZ4                                     r114         
     514574 (x 1.494)      213 MB/s  859 MB/s        70e6  284e6
LZF                                     3.6          very
     465804 (x 1.650)      143 MB/s  383 MB/s        56e6  150e6
LZO                                     2.06         1x1
     495312 (x 1.552)      242 MB/s  340 MB/s        86e6  120e6
QuickLZ                                 1.5.1b6      1
     468025 (x 1.643)      318 MB/s  274 MB/s       124e6  107e6
Shrinker                                r6           
     469750 (x 1.637)      171 MB/s  493 MB/s        66e6  191e6
Snappy                                  1.1.0        
     508506 (x 1.512)      215 MB/s  464 MB/s        72e6  157e6
ZSTD                                    0.0.1        
     335806 (x 2.289)      101 MB/s  299 MB/s        56e6  168e66
miniz                                   1.11         1
     421496 (x 1.824)     79.4 MB/s  127 MB/s        35e6   57e6
zlib                                    1.2.8        1
     367790 (x 2.090)     36.5 MB/s  183 MB/s        19e6   95e6
Codec                                   version      args
C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
done... (4*X*1) iteration(s)).

On small blocks miniz is clearly better and Shrinker is often too.
Performance on nbbs....just wow.

ADDED:

m% please ./fsbench -b65536 fast miniz,1 zlib,1 /mnt/usr/home/m/bench/nbbs.tar      
Codec                                   version      args
C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
lrrle                                   0            256
   26235632 (x 1.155)     3146 MB/s 3227 MB/s       421e6  429e6
LZ4                                     r114         
   12057544 (x 2.512)      563 MB/s 1499 MB/s       338e6  879e6
LZF                                     3.6          very
   12664450 (x 2.392)      212 MB/s  629 MB/s       123e6  360e6
LZO                                     2.06         1x1
   12382086 (x 2.446)      540 MB/s  705 MB/s       319e6  415e6
QuickLZ                                 1.5.1b6      1
   11989518 (x 2.526)      414 MB/s  480 MB/s       250e6  261e6
Shrinker                                r6           
   11488986 (x 2.636)      292 MB/s  926 MB/s       181e6  568e6
Snappy                                  1.1.0        
   12003717 (x 2.523)      507 MB/s 1013 MB/s       305e6  608e6
ZSTD                                    0.0.1        
   10269432 (x 2.949)      207 MB/s  553 MB/s       137e6  355e6
miniz                                   1.11         1
   11260776 (x 2.690)      136 MB/s  243 MB/s        85e6  123e6
zlib                                    1.2.8        1
   10512734 (x 2.881)     62.5 MB/s  298 MB/s        40e6  164e6
Codec                                   version      args
C.Size      (C.Ratio)        E.Speed   D.Speed      E.Eff. D.Eff.
done... (4*X*1) iteration(s)).

Size matters. Larger window helps a lot, but even w/out it zstd is great on this file....I'd take Shrinker though.

[gpnuma]

@m^2

Since you don't want to use git for philosophical reasons, I created a mirror of your fossil project on github (https://github.com/gpnuma/fsbench) so everybody can use it to create their codec's forks. BTW I could not see the ZSTD codebase included so I created a submodule for it.
For density integration, I created this repository : https://github.com/centaurean/fsbench-density which is a fork of the previous project.

Here are the results for enwik8 :

Codec version args
C.Size (C.Ratio) E.Speed D.Speed E.Eff. D.Eff.
density::chameleon 0.11.2
61524474 (x 1.625) 811 MB/s 1111 MB/s 312e6 427e6
density::mandala 0.11.2
53156746 (x 1.881) 455 MB/s 427 MB/s 213e6 200e6
LZ4 r114
56973116 (x 1.755) 275 MB/s 1113 MB/s 118e6 478e6
LZO 2.06 1x1
55792795 (x 1.792) 287 MB/s 372 MB/s 126e6 164e6
QuickLZ 1.5.1b6 1
52334371 (x 1.911) 281 MB/s 347 MB/s 134e6 165e6
Snappy 1.1.0
56539845 (x 1.769) 239 MB/s 774 MB/s 103e6 336e6
ZSTD 0.0.1
39573323 (x 2.527) 172 MB/s 377 MB/s 103e6 227e6
Codec version args
C.Size (C.Ratio) E.Speed D.Speed E.Eff. D.Eff.
done... (4*X*1) iteration(s)).

@Cyan

It would be really nice if you could integrate those results (or results you make up on your platform) to the ZSTD benchmark numbers. Thanks !

[m^2]

gpnuma, is it possible to download a patch from github? I can only browse it....
Thx for a hint on missing zstd code, fixed now.

[gpnuma]

gpnuma, is it possible to download a patch from github? I can only browse it....
Thx for a hint on missing zstd code, fixed now.

Sure you just need to click on the commit you want, and append a .patch to the url on github.
For example on github the density addon to fsbench is :
https://github.com/centaurean/fsbenc...133fdbba369ada

To to get a patch for this, just point your browser to the following url :
https://github.com/centaurean/fsbenc...ba369ada.patch

[Cyan]

I see there's code that would enable context preservation in zstd.c that isn't exposed to zstd.h. Would it noticeably improve small block efficiency if you exposed it to the user, possibly with some re-init function?

Yes, definitely.
Zstd is not yet friendly to small data blocks.
But the basic mechanics are there though,
so now it's mostly a matter of properly polishing the interface and expose it.

@Cyan
It would be really nice if you could integrate those results (or results you make up on your platform) to the ZSTD benchmark numbers. Thanks !

Sure !

[m^2]

ZSTD fails on this file:
http://encode.su/threads/2123-_FOSSI...2370#post42370

[Cyan]

Thanks for notification

I've tested the file linked in this thread, it seems to pass tests properly.

It could be that one of the recent changes fixed the issue.
Could you try updating the your zstd version from github ?
I just merged the "dev" branch into "master".

As a sidenote, the file downloaded seems to be different than the one benchmarked,
since its size is 442368 bytes, while the table suggests the benchmarked file should be 3471360 bytes.

[m^2]

Oh dear, something's wrong with the file indeed.

[m^2]

Updates to ZSTD have fixed the issue.

But there's another: with some compilers you #include fse, with some others you don't.
It means that a makefile has to be compiler-aware. Can't you just #include it always?

[Gonzalo]

*** stack smashing detected ***: zstd terminated

Linux binary. X86.
Reading from stdin. Writing to disk.

[Cyan]

@Gonzalo : A second fix is on the way, which I feel is related to your issue. Should be published soon within "dev" branch.

@m^2 : Good point. I've been wondering about this point for some time.
I'm even wondering if it wouldn't be better to completely integrate "fse.c", as in "copy / paste", within zstd.c
It would reduce both the problems regarding #include and number of files a client project needs to include.

The downside is that zstd.c would become very large.

[mitra]

Zstd code is very good now, I want to have zstd in such a way that it takes input from an array instead of file or stdin. This is very useful for my work of integrating Zstd compression algorithm into c-store, a column oriented database. c-store gives columns to compression algorithm in form of char*. At present, it has LZW as compression technique and I want to replace it with Zstd. So, please release a version of zstd with input being taken from char* type and output being stored in another char* variable.