Benchmarking fast LZ compressors

[dnd]

Interesting discussion, about the corpus to choose for benchmarking fast lz compressors.

Lz4 Benchmark in readme is really too misleading: https://github.com/Cyan4973/lz4/issues/109


silesia corpus as tar file: http://sun.aei.polsl.pl/~sdeor/index.php?page=silesia
size: 211,948,032 bytes.
cpu: Sandy bridge i7-2600k at 4.5 Ghz, all with gcc 5.1, ubuntu 15.04

                 size  ratio%   C MB/s     D MB/s  (bold=pareto)    MB=1.000.000
             63307408    29.9      1.53    1399.31    lzturbo 29   v1.2
             77334063    36.5      1.67    3715.44    lzturbo 19   v1.2
             77919361    36.8     36.04    1813.37    lz4 9        v1.7.0
             88463893    41.7    418.32    1616.73    lzturbo 20.1 v1.2
             93240434    44.0    395.52    1571.90    shrinker     v0.1/r9
             95093467    44.9    533.18    1822.59    lzturbo 20   v1.2
             97030389    45.8    523.45    3001.49    lzturbo 10.1 v1.2
            100018970    47.2    110.72    2899.82    yappy 1 
            100881367    47.6    567.08    1886.42    lz4 1        v1.7.0
            101195310    47.7    600.43    3183.99    lzturbo 10   v1.2
            101350919    47.8    457.18    1578.33    snappy       v1.1.2
            101835234    48.0    738.06     869.23    density 2    v0.12.5
            211948032   100.0   7900.00    7945.00    memcpy

[dnd]

I've done my own tests with the latest versions using the same Silesia corpus.
I'm using my benchmark program, all tests are repeated min. 5 times and only the best timings are considered.
I'm considering only the fastest compressors in the lz4 class (ratio < 60%) .
Quicklz, Lzo, Lzf, ... are all slower (not only on silesia) at compression and decompression.


Comparison with the lz4 benchmark at: https://github.com/Cyan4973/lz4

- lz4 is 2 times faster than snappy at decompressing ! My benchmarks are not showing such a huge difference
- lz4 is decompress at ~50% of memcpy! I can't confirm this, although memcpy speed is dependent on the cpu, ram, compiler and options used.
- The lz4 benchmark is not showing shrinker or yappy!
- but lz4 is including lzhc (lz4,9) for better compression than snappy, shrinker or yappy
- density is not included, but as this benchmark shows density has only a small advantage on compression speed with worse compression ratio.

Maybe someone can redo the tests from the lz4 page on a recent desktop cpu.
Personally, i don't think that Silesia is not appropriate for this sort of speed benchmarks.
Not too small and big enough avoiding pure cache benchmarks.

P.S. Yappy is crashing on some files under windows