Results 1 to 2 of 2

Thread: Benchmarking fast LZ compressors

  1. #1
    Member
    Join Date
    Mar 2013
    Location
    Worldwide
    Posts
    565
    Thanks
    67
    Thanked 198 Times in 147 Posts

    Benchmarking fast LZ compressors

    Interesting discussion, about the corpus to choose for benchmarking fast lz compressors.

    Lz4 Benchmark in readme is really too misleading: https://github.com/Cyan4973/lz4/issues/109


    silesia corpus as tar file: http://sun.aei.polsl.pl/~sdeor/index.php?page=silesia
    size: 211,948,032 bytes.
    cpu: Sandy bridge i7-2600k at 4.5 Ghz, all with gcc 5.1, ubuntu 15.04

    Code:
                     size  ratio%   C MB/s     D MB/s  (bold=pareto)    MB=1.000.000
                 63307408    29.9      1.53    1399.31    lzturbo 29   v1.2
                 77334063    36.5      1.67    3715.44    lzturbo 19   v1.2
                 77919361    36.8     36.04    1813.37    lz4 9        v1.7.0
                 88463893    41.7    418.32    1616.73    lzturbo 20.1 v1.2
                 93240434    44.0    395.52    1571.90    shrinker     v0.1/r9
                 95093467    44.9    533.18    1822.59    lzturbo 20   v1.2
                 97030389    45.8    523.45    3001.49    lzturbo 10.1 v1.2
                100018970    47.2    110.72    2899.82    yappy 1 
                100881367    47.6    567.08    1886.42    lz4 1        v1.7.0
                101195310    47.7    600.43    3183.99    lzturbo 10   v1.2
                101350919    47.8    457.18    1578.33    snappy       v1.1.2
                101835234    48.0    738.06     869.23    density 2    v0.12.5
                211948032   100.0   7900.00    7945.00    memcpy

    Last edited by dnd; 15th June 2015 at 18:47.

  2. #2
    Member
    Join Date
    Mar 2013
    Location
    Worldwide
    Posts
    565
    Thanks
    67
    Thanked 198 Times in 147 Posts
    I've done my own tests with the latest versions using the same Silesia corpus.
    I'm using my benchmark program, all tests are repeated min. 5 times and only the best timings are considered.
    I'm considering only the fastest compressors in the lz4 class (ratio < 60%) .
    Quicklz, Lzo, Lzf, ... are all slower (not only on silesia) at compression and decompression.


    Comparison with the lz4 benchmark at: https://github.com/Cyan4973/lz4

    - lz4 is 2 times faster than snappy at decompressing ! My benchmarks are not showing such a huge difference
    - lz4 is decompress at ~50% of memcpy! I can't confirm this, although memcpy speed is dependent on the cpu, ram, compiler and options used.
    - The lz4 benchmark is not showing shrinker or yappy!
    - but lz4 is including lzhc (lz4,9) for better compression than snappy, shrinker or yappy
    - density is not included, but as this benchmark shows density has only a small advantage on compression speed with worse compression ratio.

    Maybe someone can redo the tests from the lz4 page on a recent desktop cpu.
    Personally, i don't think that Silesia is not appropriate for this sort of speed benchmarks.
    Not too small and big enough avoiding pure cache benchmarks.

    P.S. Yappy is crashing on some files under windows
    Last edited by dnd; 15th June 2015 at 20:22.

Similar Threads

  1. Benchmarking Entropy Coders
    By dnd in forum Data Compression
    Replies: 194
    Last Post: 22nd February 2020, 00:02
  2. Archiver benchmarking framework
    By Bulat Ziganshin in forum Data Compression
    Replies: 11
    Last Post: 8th February 2013, 19:46
  3. Compression benchmarking: 64 bit images and 24 bit codecs
    By m^2 in forum The Off-Topic Lounge
    Replies: 6
    Last Post: 30th November 2011, 16:01
  4. Some of my toy compressors
    By RichSelian in forum Data Compression
    Replies: 27
    Last Post: 6th October 2011, 05:09
  5. paq8f w/ .DXEs (DJGPPv2, DOS, benchmarking)
    By Rugxulo in forum Data Compression
    Replies: 4
    Last Post: 2nd February 2010, 14:32

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •