Results 1 to 19 of 19

Thread: SHARC / DENSITY news and updates

  1. #1
    Member
    Join Date
    Aug 2013
    Location
    France
    Posts
    77
    Thanks
    27
    Thanked 26 Times in 11 Posts

    SHARC / DENSITY news and updates

    Hello all,

    I just thought I'd create a thread for this instead of re-creating new ones every time.
    A quick introduction for those (numerous ) who don't know SHARC (or DENSITY) : it's a super fast compression utility based on the DENSITY bsd library. The top priority of the library is speed, and of course getting the best possible compression ratio at high velocities.

    Everything is open source and the sources are available here (SHARC) : https://github.com/centaurean/sharc and here (DENSITY) : https://github.com/centaurean/density.
    SHARC binaries for Linux, OSX and Windows can be downloaded here https://github.com/centaurean/sharc/releases.

    A new release of both the compression client (SHARC) and the library (DENSITY) was published yesterday : SHARC 0.9.11 beta / DENSITY 0.9.12 beta.
    Its main features are the following :
    - Inclusion of a new state of the art algorithm (heavily optimized mixed hashes / predictions) in cooperation with Piotr Tarsa of this forum for the -c2 mode
    - Speed and compression ratio improvements in all modes

    LZ4 being the fastest compressor I know of, this is a quick comparison with LZ4 (r109) on my testing platform (Core i7 2.3 GHz, OSX, ramdisk) of both SHARC compression modes using the famous enwik8 file. Both projects were compiled using the same compiler (GCC 4., the latest source codes available and their standard makefiles.

    LZ4 r109 default mode
    $ time ./lz4c -y enwik8
    Compressed filename will be : enwik8.lz4


    Compressed 100000000 bytes into 56995506 bytes ==> 57.00%


    real 0m0.492s
    user 0m0.383s
    sys 0m0.107s

    $ time ./lz4c -d -y enwik8.lz4
    Decoding file enwik8
    Successfully decoded 100000000 bytes


    real 0m0.258s
    user 0m0.093s
    sys 0m0.158s
    SHARC 0.9.11 -c1
    $ time ./sharc -n -c1 enwik8
    Compressed enwik8 (100,000,000 bytes) to enwik8.sharc (61,611,730 bytes), Ratio out / in = 61.6%, Time = 0.214 s, Speed = 446 MB/s


    real 0m0.222s
    user 0m0.115s
    sys 0m0.105s

    $ time ./sharc -n -d enwik8.sharc

    Decompressed enwik8.sharc (61,611,730 bytes) to enwik8 (100,000,000 bytes), Time = 0.230 s, Speed = 414 MB/s


    real 0m0.244s
    user 0m0.096s
    sys 0m0.143s
    SHARC 0.9.11 -c2
    $ time ./sharc -n -c2 enwik8
    Compressed enwik8 (100,000,000 bytes) to enwik8.sharc (53,175,042 bytes), Ratio out / in = 53.2%, Time = 0.290 s, Speed = 328 MB/s


    real 0m0.300s
    user 0m0.202s
    sys 0m0.095s


    $ time ./sharc -n -d enwik8.sharc
    Decompressed enwik8.sharc (53,175,042 bytes) to enwik8 (100,000,000 bytes), Time = 0.351 s, Speed = 272 MB/s


    real 0m0.365s
    user 0m0.224s
    sys 0m0.134s
    More benchmarks are to come ! Thanks for your time.
    Last edited by gpnuma; 13th December 2013 at 19:09.

  2. Thanks (2):

    Euph0ria (14th December 2013),Piotr Tarsa (13th December 2013)

  3. #2
    Member
    Join Date
    Jun 2009
    Location
    Kraków, Poland
    Posts
    1,498
    Thanks
    26
    Thanked 135 Times in 103 Posts
    One compressor from LTCB drew my attention: stz by Bruno Wyttenbach. It's extremely fast, yet it has superb compression ratio for that speed. For my taste its efficiency on enwik9 beats both SHARC and LZ4. Too bad the download link on LTCB is broken. Same situation with slug 1.1b by Christian Martelock.

  4. #3
    Member
    Join Date
    Aug 2013
    Location
    France
    Posts
    77
    Thanks
    27
    Thanked 26 Times in 11 Posts
    Hey Piotr !

    I found the first one (STZ) here : http://phantasie.tonempire.net/t125-...eed-compressor .

    I did a quick test on windows 8 and SHARC -c1 is 4x faster at least, SHARC -c2 3x faster ... strange.

  5. #4
    Member
    Join Date
    Jun 2009
    Location
    Kraków, Poland
    Posts
    1,498
    Thanks
    26
    Thanked 135 Times in 103 Posts
    Have you measured process times or wall times? Maybe it has inefficient I/O.

  6. #5
    Member
    Join Date
    Aug 2013
    Location
    France
    Posts
    77
    Thanks
    27
    Thanked 26 Times in 11 Posts
    No.. I can do if you know a good equivalent of "time" on windows ?

  7. #6
    Member
    Join Date
    Jun 2009
    Location
    Kraków, Poland
    Posts
    1,498
    Thanks
    26
    Thanked 135 Times in 103 Posts

  8. #7
    Tester
    Black_Fox's Avatar
    Join Date
    May 2008
    Location
    [CZE] Czechia
    Posts
    471
    Thanks
    26
    Thanked 9 Times in 8 Posts
    1st one is traditional, 2nd one is this forum's rising star
    1) http://encode.su/threads/245-timer301-zip
    2) http://encode.su/threads/1838-Comman...Profiling-Tool

    EDIT: Piotr beat me to it
    I am... Black_Fox... my discontinued benchmark
    "No one involved in computers would ever say that a certain amount of memory is enough for all time? I keep bumping into that silly quotation attributed to me that says 640K of memory is enough. There's never a citation; the quotation just floats like a rumor, repeated again and again." -- Bill Gates

  9. #8
    Member
    Join Date
    Aug 2013
    Location
    France
    Posts
    77
    Thanks
    27
    Thanked 26 Times in 11 Posts
    Thanks This is the result (the machine is heavily loaded so not at peak performance) :

    SHARC 0.9.11 -c1

    PS E:\> .\ProcProfile64.exe .\sharc_win64.exe -n .\enwik8
    Compressed .\enwik8 (100,000,000 bytes) to .\enwik8.sharc (61,611,730 bytes), Ratio out / in = 61.6%, Time = 0.210 s, Sp
    eed = 454 MB/s


    Process ID : 712
    Thread ID : 508
    Process Exit Code: 1
    Thread Exit Code : 259


    User Time : 0.187s
    Kernel Time : 0.031s
    Process Time : 0.218s
    Clock Time : 0.237s


    Working Set : 3552 KB
    Paged Pool : 23 KB
    Nonpaged Pool : 3 KB
    Pagefile : 2664 KB
    Page Fault Count : 905


    IO Read : 97656 KB (in 192 reads )
    IO Write : 60167 KB (in 251 writes)
    IO Other : 0 KB (in 24 others)
    SHARC 0.9.11 -c2
    PS E:\> .\ProcProfile64.exe .\sharc_win64.exe -n -c2 .\enwik8
    Compressed .\enwik8 (100,000,000 bytes) to .\enwik8.sharc (53,175,042 bytes), Ratio out / in = 53.2%, Time = 0.309 s, Sp
    eed = 308 MB/s


    Process ID : 6108
    Thread ID : 588
    Process Exit Code: 1
    Thread Exit Code : 1


    User Time : 0.281s
    Kernel Time : 0.031s
    Process Time : 0.312s
    Clock Time : 0.326s


    Working Set : 4576 KB
    Paged Pool : 23 KB
    Nonpaged Pool : 3 KB
    Pagefile : 3176 KB
    Page Fault Count : 1161


    IO Read : 97656 KB (in 192 reads )
    IO Write : 51928 KB (in 219 writes)
    IO Other : 0 KB (in 24 others)
    STZ.exe
    PS E:\> .\ProcProfile64.exe .\stz.exe .\enwik8******** stz v0.8.3 by Bruno Wyttenbach - Mar 19 2011 ********
    Compressing ...
    File size : 95.4Mb -> 47.8Mb (50.14%)
    Working time : 0.81s at 117.9Mb/s


    Process ID : 5996
    Thread ID : 5144
    Process Exit Code: 0
    Thread Exit Code : 0


    User Time : 0.578s
    Kernel Time : 0.062s
    Process Time : 0.640s
    Clock Time : 0.828s


    Working Set : 5140 KB
    Paged Pool : 19 KB
    Nonpaged Pool : 4 KB
    Pagefile : 5068 KB
    Page Fault Count : 1305


    IO Read : 97656 KB (in 50 reads )
    IO Write : 48968 KB (in 205 writes)
    IO Other : 0 KB (in 129 others)
    Last edited by gpnuma; 13th December 2013 at 21:52.

  10. #9
    Member
    Join Date
    Jun 2009
    Location
    Kraków, Poland
    Posts
    1,498
    Thanks
    26
    Thanked 135 Times in 103 Posts
    STZ has other modes too. LZBW2A was shown on LTCB to perform well on enwik9. Although testing on one file is prone to give non-objective conclusions.

  11. #10
    Member
    Join Date
    Aug 2013
    Location
    France
    Posts
    77
    Thanks
    27
    Thanked 26 Times in 11 Posts
    Yes but the default is the fastest compression (I tried using the -c switch it's the same as no switch) :

    E:\>stz.exe -h
    ******** stz v0.8.3 by Bruno Wyttenbach - Mar 19 2011 ********


    High speed compressor.


    Use : stz [-h] [-cu] FileIn [FileOut]
    -c Compress with LZBW2 algorithm (Best compression speed)
    -c1 Compress with LZBW3 algorithm
    -c2 Compress with LZBW2A algorithm (Best compression)
    -c3 Compress with LZBW3A algorithm
    -c4 Compress with LZBW2B algorithm (Experimental)
    -c5 Compress with LZBW3B algorithm
    -u Uncompress
    -h Help


    Examples : stz -c3 file
    stz -u file.stz
    stz file file2.stz
    Drag'n'Drop file onto the compressor.


    Please report any bug at bwy@free.fr
    I wonder if Matt can give us the actual timings on his platform, instead of a ns/byte measurement ?

  12. #11
    Member
    Join Date
    Jun 2009
    Location
    Kraków, Poland
    Posts
    1,498
    Thanks
    26
    Thanked 135 Times in 103 Posts
    Actually, ns/ B is equal to s/ GB ;] Matt reports process times for fast compressors and wall times for others.

    And on Matt's system on LTCB stz -c2 was actually the fastest, compression speed wise. Maybe that was just luck and overall the default is much faster, but I haven't tested.

  13. #12
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,257
    Thanks
    307
    Thanked 797 Times in 489 Posts
    Yes, I use process time or wall time, whichever is faster. For single threaded programs it is normally process time. For very fast programs there is a big difference due to the disk being much slower. It is in seconds for enwik9. I don't use fractions of a second because the margin of error is usually more than 1 second. Also, tests might be on different machines (see note column), so it is a rough approximation.

  14. #13
    Member
    Join Date
    Aug 2013
    Location
    France
    Posts
    77
    Thanks
    27
    Thanked 26 Times in 11 Posts
    Quote Originally Posted by Matt Mahoney View Post
    Yes, I use process time or wall time, whichever is faster. For single threaded programs it is normally process time. For very fast programs there is a big difference due to the disk being much slower. It is in seconds for enwik9. I don't use fractions of a second because the margin of error is usually more than 1 second. Also, tests might be on different machines (see note column), so it is a rough approximation.
    Edit : Sorry I re-read your previous message Matt and I understand your benchmark is targeted at compression ratio evaluation and not speed measurement although you display it for the user's benefit.
    Last edited by gpnuma; 14th December 2013 at 23:53.

  15. #14
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,610
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by gpnuma View Post
    LZ4 being the fastest compressor I know of, this is a quick comparison with LZ4 (r109) on my testing platform (Core i7 2.3 GHz, OSX, ramdisk) of both SHARC compression modes using the famous enwik8 file. Both projects were compiled using the same compiler (GCC 4., the latest source codes available and their standard makefiles.
    RLE64 is much faster than LZ4. Iit's not quite in the same league when it comes to ratio, though it depends on what you compress.
    http://sourceforge.net/projects/nikkhokkho/

  16. #15
    Member
    Join Date
    Aug 2013
    Location
    France
    Posts
    77
    Thanks
    27
    Thanked 26 Times in 11 Posts
    Thanks m^2, I just tried RLE64 on enwik8, compression ratio = 99.99 % ... For enwik9 it's 99.99 % as well. I suppose it's tailored for use on RLE-easy-to-compress files like BMP images.
    I'm not sure what the timings displayed stand for, but ProcProfile gives the following infos :

    PS E:\> .\ProcProfile64.exe .\RLE64.exe 64 .\enwik8 .\enwik8.rle64 .\enwik8.unrle64


    RLE64 R3.10 (c) 2011-2013 by Javier Gutierrez Chamorro
    http://nikkhokkho.sourceforge.net/static.php?page=RLE64
    A fast Run Length Encoding (RLE) compressor / decompressor.


    Compressed .\enwik8 in 42 ms (2380 MiB/s).
    Uncompressed .\enwik8.rle64 in 36 ms (2777 MiB/s).


    Process ID : 5680
    Thread ID : 96
    Process Exit Code: 0
    Thread Exit Code : 0


    User Time : 0.062s
    Kernel Time : 0.250s
    Process Time : 0.312s
    Clock Time : 2.141s


    Working Set : 197436 KB
    Paged Pool : 15 KB
    Nonpaged Pool : 3 KB
    Pagefile : 293816 KB
    Page Fault Count : 73820


    IO Read : 97656 KB (in 2 reads )
    IO Write : 195308 KB (in 11 writes)
    IO Other : 0 KB (in 18 others)


    PS E:\> ls -l .\enwik8.rle64




    Répertoire : E:\




    Mode LastWriteTime Length Name
    ---- ------------- ------ ----
    -a--- 14/12/2013 22:04 99995432 enwik8.rle64
    Last edited by gpnuma; 15th December 2013 at 00:12.

  17. #16
    Tester
    Stephan Busch's Avatar
    Join Date
    May 2008
    Location
    Bremen, Germany
    Posts
    876
    Thanks
    474
    Thanked 175 Times in 85 Posts
    Does anyone have a win32 compile of latest Density 0.12.5?

  18. #17
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    309
    Thanks
    68
    Thanked 173 Times in 64 Posts
    I think I found a bug in density 0.12.5 beta level 3.

    When I compress the attached file (65536 bytes) decompressor returns 65532 bytes written.
    Attached Files Attached Files

  19. #18
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,610
    Thanks
    30
    Thanked 65 Times in 47 Posts
    I also have a number of errors to report.
    Try to compile sharc with the attached makefile and decompress the input files to see a variety of crashes.
    Attached Files Attached Files

  20. #19
    Member
    Join Date
    Feb 2018
    Location
    Czechia
    Posts
    2
    Thanks
    0
    Thanked 0 Times in 0 Posts

    correction

    pls would you be so kind and send me a link to download sharc windows binaries ? I am not able to build it and link https://github.com/centaurean/sharc/releases does not work.

    There are sharc_win32.exe and sharc_win64.exe

Similar Threads

  1. zpaq updates
    By Matt Mahoney in forum Data Compression
    Replies: 2618
    Last Post: Today, 01:36
  2. LIBSSC : SHARC's algorithms unleashed
    By gpnuma in forum Data Compression
    Replies: 131
    Last Post: 18th April 2014, 22:19
  3. GUI winzpaq updates
    By Sportman in forum Data Compression
    Replies: 31
    Last Post: 4th August 2013, 00:21
  4. comprox updates
    By RichSelian in forum Data Compression
    Replies: 1
    Last Post: 6th November 2011, 00:19
  5. Metacompressor.com benchmark updates
    By Sportman in forum Data Compression
    Replies: 79
    Last Post: 22nd April 2009, 04:24

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •