Page 4 of 4 FirstFirst ... 234
Results 91 to 110 of 110

Thread: In-memory benchmark with fastest LZSS (QuickLZ, Snappy) compressors

  1. #91
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    309
    Thanks
    68
    Thanked 173 Times in 64 Posts
    Quote Originally Posted by cbloom View Post
    Awesome project, thanks. Some annoying requests which maybe you also think would be good for the project :

    (I'd do it myself unfortunately I can't build gcc code for Windows and the whole advantage of lzbench for me is that it compiles lots of projects that don't build in MSVC)

    1. I'd like this to be optional :

    #ifdef WINDOWS
    SetPriorityClass(GetCurrentProcess(), REALTIME_PRIORITY_CLASS);
    #else
    setpriority(PRIO_PROCESS, 0, -20);
    #endif

    I'm not sure how it's affecting speeds exactly and I prefer to do runs in the same way that client apps will. (eg. clients won't be doing that so I shouldn't either)

    2. I'd like an option (or just always) to invalidate the cache between runs, so the timing is reflective of un-primed-cache performance. This can make a huge difference on fast compressors on files that fit in cache.

    3. An option to show times instead of speeds (eg. instead of 1000 MB/s report seconds or millis) would be nice

    4. Why use QPC instead of TSC on windows? Probably not a big deal but I think TSC is probably just better now that modern chips make it stable. It's harder to get the frequency I guess.

    5. In the list of compressors made with -l , log the range of levels after the name, something like

    lz5hc 1.4.1 [1-15]

    6. Keep up the awesome work!
    Thanks for your feedback. I will try to include your remarks in the next version. If there will be a greater demand I can prepare lzbench to compile with Visual Studio (at least with most of compressors).
    Last edited by inikep; 11th August 2016 at 14:46.

  2. #92
    Member lz77's Avatar
    Join Date
    Jan 2016
    Location
    Russia
    Posts
    176
    Thanks
    60
    Thanked 16 Times in 12 Posts
    Quote Originally Posted by inikep View Post
    The input file (100 MB) is a concatenation of 10 different files, about 10 MB each: bmp, dct_coeffs, english_dic, ENWIK, exe, fp_log, hlp, XML, pdf, ncb.
    To get highest results you should "warm" the core before benchmark
    Code:
    SetProcessAffinityMask(GetCurrentProcess(), 1);
    	volatile int zomg = 1;
    	for ( int i=1; i<1000000000; i++ )
    		zomg *= i;
    Read about this https://habrahabr.ru/post/113682/ (sorry, in Russian).
    Could you please upload your input file? I want to download it to test my LZ77 examples, thanks

    Serge

  3. Thanks (2):

    Bulat Ziganshin (12th August 2016),ne0n (27th June 2016)

  4. #93
    Member
    Join Date
    Jan 2014
    Location
    Bothell, Washington, USA
    Posts
    697
    Thanks
    154
    Thanked 186 Times in 109 Posts
    Hi Inikep,

    I really like lzbench. It is a very nice tool for comparing LZ compression performance.

    Could you possibly add GLZA? Modified lzbench and new GLZA source code is attached. Also, I wasn't able to get the "-eall" option to test more than one token's worth of files, so beware I put in a hack to get around that problem.

    Here's a sample of the top 20 results on 1musk10.txt, sorted by compression ratio:

    Code:
    The results sorted by column number 5:
    Compressor name         Compress. Decompress. Compr. size  Ratio Filename
    glza 0.7.1               0.16 MB/s    25 MB/s      309745  23.03 1musk10.txt
    csc 3.3 -5               3.27 MB/s    35 MB/s      375672  27.94 1musk10.txt
    brotli 0.4.0 -11         0.37 MB/s   234 MB/s      382317  28.43 1musk10.txt
    lzlib 1.7 -6             1.51 MB/s    39 MB/s      385331  28.65 1musk10.txt
    lzlib 1.7 -9             1.54 MB/s    39 MB/s      386360  28.73 1musk10.txt
    lzma 9.38 -5             1.58 MB/s    59 MB/s      386490  28.74 1musk10.txt
    csc 3.3 -3               4.17 MB/s    34 MB/s      386589  28.75 1musk10.txt
    xz 5.2.2 -6              1.70 MB/s    54 MB/s      386689  28.76 1musk10.txt
    xz 5.2.2 -9              1.65 MB/s    53 MB/s      386689  28.76 1musk10.txt
    zstd 0.8.0 -22           1.82 MB/s   397 MB/s      392000  29.15 1musk10.txt
    zstd 0.8.0 -18           2.22 MB/s   404 MB/s      392716  29.20 1musk10.txt
    tornado 0.6a -16         1.81 MB/s   124 MB/s      392901  29.22 1musk10.txt
    tornado 0.6a -13         3.38 MB/s   116 MB/s      411640  30.61 1musk10.txt
    zstd 0.8.0 -15           3.49 MB/s   412 MB/s      413727  30.77 1musk10.txt
    lzham 1.0 -d26 -1        1.58 MB/s   140 MB/s      417559  31.05 1musk10.txt
    zling 2016-01-10 -4        20 MB/s    83 MB/s      417749  31.07 1musk10.txt
    csc 3.3 -1                 11 MB/s    34 MB/s      418998  31.16 1musk10.txt
    zling 2016-01-10 -3        23 MB/s    83 MB/s      421626  31.35 1musk10.txt
    tornado 0.6a -10         4.53 MB/s   121 MB/s      421810  31.37 1musk10.txt
    brotli 0.4.0 -8          5.93 MB/s   244 MB/s      425235  31.62 1musk10.txt
    Edit: I fixed the table and replace the code because I found a memory initialization problem that shows up when a file is repeatedly decoded. There seems to be another bug, glza runs fine but on files over 1 - 3 MB it causes the next program to crash. The cause is not obvious; I checked memory pointers and they look okay but the workmem address seems to be larger. I don't understand much of lzbench code, so I'm not sure what is going on.
    Last edited by Kennon Conrad; 14th August 2016 at 22:36.

  5. Thanks:

    Stephan Busch (12th August 2016)

  6. #94
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,593
    Thanks
    801
    Thanked 698 Times in 378 Posts
    Use CODE tag, Luke!
    Last edited by Bulat Ziganshin; 12th August 2016 at 15:12.

  7. Thanks:

    Kennon Conrad (12th August 2016)

  8. #95
    Tester
    Stephan Busch's Avatar
    Join Date
    May 2008
    Location
    Bremen, Germany
    Posts
    876
    Thanks
    474
    Thanked 175 Times in 85 Posts
    can anybody please compile it?

  9. #96
    Member
    Join Date
    Jan 2014
    Location
    Bothell, Washington, USA
    Posts
    697
    Thanks
    154
    Thanked 186 Times in 109 Posts
    Quote Originally Posted by Stephan Busch View Post
    can anybody please compile it?
    Yes. I fixed the problem with GLZA causing other compressors to crash and included a win64 executable this time.

    If anyone else wants to compile lzbench with this code, you need to first download the lzbench source code and then replace a few of those and add the glza directory in this attachment. Then run the Makefile and that should do it (assuming minGW and Posix threads are installed).
    Last edited by Kennon Conrad; 14th August 2016 at 22:35.

  10. Thanks:

    Stephan Busch (12th August 2016)

  11. #97
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    309
    Thanks
    68
    Thanked 173 Times in 64 Posts
    I've added glza to lzbench (https://github.com/inikep/lzbench/commits/dev) but still testing

    1. glza doesn't work with gcc older than 4.9 because of:
    Code:
    glza/GLZAcompress.c:37:23: fatal error: stdatomic.h: No such file or directory
     #include <stdatomic.h>
    2. glza uses many threads and with lzbench's SetPriorityClass/setpriority a computer loses responsiveness therefore there is a new switch:
    Code:
    -r = disable real-time process priority
    3. still there is a bug after lzbench-master_glza_prototypeB.zip
    Code:
    FATAL ERROR - dictionary malloc failure
    To generate: "lzbench.exe -t0 -u0 -j1000 -r -eglza NEWS" (NEWS is from main directory)
    It means 1 compression and 1000 decompression loops.
    or "lzbench.exe -r -u2 -eglza NEWS" what means 2 seconds for decompression loops.

  12. Thanks:

    Kennon Conrad (12th August 2016)

  13. #98
    Member
    Join Date
    Jan 2014
    Location
    Bothell, Washington, USA
    Posts
    697
    Thanks
    154
    Thanked 186 Times in 109 Posts
    Quote Originally Posted by inikep View Post
    1. glza doesn't work with gcc older than 4.9 because of:
    Code:
    glza/GLZAcompress.c:37:23: fatal error: stdatomic.h: No such file or directory
     #include <stdatomic.h>
    Is this a problem? Do you know of any simple alternatives?

    Quote Originally Posted by inikep View Post
    2. glza uses many threads and with lzbench's SetPriorityClass/setpriority a computer loses responsiveness therefore there is a new switch:
    Code:
    lzbench crashes at the end of all testing on NEWS with -eall.  It seems to be glza related but I don't know the cause yet.
    -r = disable real-time process priority
    An area glza could use some work in....

    Quote Originally Posted by inikep View Post
    3. still there is a bug after lzbench-master_glza_prototypeB.zip
    Code:
    FATAL ERROR - dictionary malloc failure
    To generate: "lzbench.exe -t0 -u0 -j1000 -r -eglza NEWS" (NEWS is from main directory)
    It means 1 compression and 1000 decompression loops.
    or "lzbench.exe -r -u2 -eglza NEWS" what means 2 seconds for decompression loops.
    Fixed in lzbench-master_glza_prototypeD, see attached. The glza files and the glza section in compressors.cpp were modified.
    Last edited by Kennon Conrad; 14th August 2016 at 22:35. Reason: Fixed bug

  14. #99
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    309
    Thanks
    68
    Thanked 173 Times in 64 Posts
    Quote Originally Posted by Kennon Conrad View Post
    Is this a problem?
    All other compressors except LZSSE (which requires -msse4.1 not available with gcc 4.6) work with gcc 4.6 and gcc 4.8.


    Quote Originally Posted by Kennon Conrad View Post
    Fixed in lzbench-master_glza_prototypeD
    It seems to work OK.

  15. #100
    Member jibz's Avatar
    Join Date
    Jan 2015
    Location
    Denmark
    Posts
    124
    Thanks
    106
    Thanked 71 Times in 51 Posts
    Quote Originally Posted by Kennon Conrad View Post
    Is this a problem? Do you know of any simple alternatives?
    Perhaps https://tinycthread.github.io/ might be an option, depending on your needs.

  16. Thanks:

    Kennon Conrad (13th August 2016)

  17. #101
    Member
    Join Date
    Jan 2014
    Location
    Bothell, Washington, USA
    Posts
    697
    Thanks
    154
    Thanked 186 Times in 109 Posts
    Gosh, that's the one thing on a list of things Nemequ gave me to think about that I haven't gotten around to yet. I though it was for multi-threading only and didn't realize it provides atomics (apparently?). I'll have to take a look.

  18. #102
    Member
    Join Date
    Jan 2014
    Location
    Bothell, Washington, USA
    Posts
    697
    Thanks
    154
    Thanked 186 Times in 109 Posts
    Quote Originally Posted by inikep View Post
    It seems to work OK.
    Thanks for adding and testing. I did find one additional problem and replaced compressors.cpp in the prototypeD file. For files that are delta encoded GLZA modifies the input so now the glza section in compressors.cpp creates a local copy of the input to give GLZA. It looks like maybe this should be done through an init function and use workmem, but I'm not exactly sure about the intent. Also, is there something that should be done to limit GLZA to blocks of 2 GB?

  19. #103
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    309
    Thanks
    68
    Thanked 173 Times in 64 Posts
    Quote Originally Posted by Kennon Conrad View Post
    For files that are delta encoded GLZA modifies the input
    This is unusual and unexpected. You should fix it or give a huge warning next to the function declaration.

    Moreover I think you should change your functions from
    GLZAformat(insize, inbuf, &outsize);
    to
    GLZAformat(insize, inbuf, &outsize, outbuf);
    which will allow to provide an external output buffer. I think all other compressors do that.
    You should allocate the buffer only when outbuf==NULL or outsize==0.
    Currently there is too many unnecessary malloc() and free() calls.

  20. #104
    Member
    Join Date
    Jan 2014
    Location
    Bothell, Washington, USA
    Posts
    697
    Thanks
    154
    Thanked 186 Times in 109 Posts
    Quote Originally Posted by inikep View Post
    This is unusual and unexpected. You should fix it or give a huge warning next to the function declaration.
    It wasn't bad when input was never reused, always read from disk. Integrating glza into lzbench has pointed out lots of things that needed to be changed so it has been a very good exercise.

    Quote Originally Posted by inikep View Post
    Moreover I think you should change your functions from
    GLZAformat(insize, inbuf, &outsize);
    to
    GLZAformat(insize, inbuf, &outsize, outbuf);
    which will allow to provide an external output buffer. I think all other compressors do that.
    You should allocate the buffer only when outbuf==NULL or outsize==0.
    Currently there is too many unnecessary malloc() and free() calls.
    Would this be reasonable?:

    Code:
    int64_t lzbench_glza_compress(char *inbuf, size_t insize, char *outbuf, size_t outsize, size_t, size_t, char*)
    {
        outbuf = (char *)GLZAcomp(insize, (uint8_t *)inbuf, &outsize, (uint8_t *)outbuf);
        return outsize;
    }
    I have this working and it seems like it would be most stable for lzbench, allowing glza to handle any temporary ugliness and be improved over time without impacting the lzbench code. On the bad side, the malloc and free calls still exist, but now they are in the glza code instead of the lzbench code. If you think these need to be eliminated, I could add the workmem parameter. I was thinking it wasn't a big deal because compression is already slow but maybe there is more to it than I am aware of.

  21. #105
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    309
    Thanks
    68
    Thanked 173 Times in 64 Posts
    Quote Originally Posted by Kennon Conrad View Post
    Would this be reasonable?
    It's fine for me.

    But please remember about checking if memory was allocated because in your code I found some missing checks:

    Code:
    int64_t lzbench_glza_compress(char *inbuf, size_t insize, char *outbuf, size_t outsize, size_t, size_t, char*)
    {
    	char * tempbuf = (char *)malloc(insize);
            if (!tempbuf) return 0;
    	memcpy(tempbuf, inbuf, insize);
    	inbuf = (char *)GLZAformat(insize, (uint8_t *)tempbuf, &outsize);
    	insize = outsize;
    	free(tempbuf);
    	tempbuf = (char *)GLZAcompress(insize, (uint8_t *)inbuf, &outsize);
            free(inbuf);
            if (!tempbuf) return 0;	
    	inbuf = tempbuf;
    	insize = outsize;
    	outbuf = (char *)GLZAencode(insize, (uint8_t *)inbuf, &outsize, (uint8_t *)outbuf, (FILE *)0);
    	free(inbuf);
            if (!outbuf) return 0;
    	return outsize;
    }
    Quote Originally Posted by Kennon Conrad View Post
    If you think these need to be eliminated, I could add the workmem parameter. I was thinking it wasn't a big deal because compression is already slow but maybe there is more to it than I am aware of.
    When you are compressing small files it doesn't matter but when we use 1 GB+ files many malloc() and free() can be a problem as it's hard to allocate continuous blocks of memory of this size.

  22. #106
    Member
    Join Date
    Jan 2014
    Location
    Bothell, Washington, USA
    Posts
    697
    Thanks
    154
    Thanked 186 Times in 109 Posts
    Thanks for the explanation. GLZA uses a lot of memory for compression so if it fails on the I/O malloc it's probably just as well, at least for now.

    I added the malloc checks in the GLZA code and changed other exits in the GLZA routines to return a failure (so other programs will still run in lzbench).

    Now the compressors.cpp code is this:

    Code:
    #ifndef BENCH_REMOVE_GLZA
    #include "glza/GLZAcomp.h"
    #include "glza/GLZAdecode.h"
    
    int64_t lzbench_glza_compress(char *inbuf, size_t insize, char *outbuf, size_t outsize, size_t, size_t, char*)
    {
        if (GLZAcomp(insize, (uint8_t *)inbuf, &outsize, (uint8_t *)outbuf, (FILE *)0) == 0) return(0);
        return outsize;
    }
    
    int64_t lzbench_glza_decompress(char *inbuf, size_t insize, char *outbuf, size_t outsize, size_t, size_t, char*)
    {
        if (GLZAdecode(insize, (uint8_t *)inbuf, &outsize, (uint8_t *)outbuf, (FILE *)0) == 0) return(0);
        return outsize;
    }
    
    #endif
    and GLZAcomp.o has to be added to the Makefile and the GLZA code replaced.
    Last edited by Kennon Conrad; 14th August 2016 at 22:34.

  23. #107
    Member
    Join Date
    Jan 2014
    Location
    Bothell, Washington, USA
    Posts
    697
    Thanks
    154
    Thanked 186 Times in 109 Posts
    Did some more testing and found I added a bug when taking out the exit's. Aarrgghh! "Final" version attached and removing others.
    Attached Files Attached Files

  24. Thanks:

    inikep (15th August 2016)

  25. #108
    Member
    Join Date
    Aug 2016
    Location
    US
    Posts
    1
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by inikep View Post
    Thanks for your feedback. I will try to include your remarks in the next version. If there will be a greater demand I can prepare lzbench to compile with Visual Studio (at least with most of compressors).
    +1

    I'd really love to have the ability to compile using Visual Studio! Thanks for all your work!

  26. #109
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    309
    Thanks
    68
    Thanked 173 Times in 64 Posts
    It's already on my "to do" list. I sure some compressors will have to be disabled because it will be too time consuming to fix them for Visual Studio. So far I can recommend MinGW for Windows.

  27. #110
    Member
    Join Date
    Jul 2013
    Location
    United States
    Posts
    194
    Thanks
    44
    Thanked 140 Times in 69 Posts
    FWIW, as of about 2 weeks ago all the codecs which are enabled by default in Squash work with VS 12+ (mostly thanks to Jørgen Ibsen, who has submitted lots of fixes to various codecs), so it should be relatively straightforward to get them working for you. We also have Windows CI builds set up on AppVeyor, and we've already helped a few other projects (like Brotli and LZFSE) do the same, so hopefully we'll notice any future breakage pretty quickly.

    IIRC Density is the only somewhat interesting codec which doesn't work on Windows, but we've had to disable it for security reasons (hence "enabled by default").

  28. Thanks:

    inikep (30th August 2016)

Page 4 of 4 FirstFirst ... 234

Similar Threads

  1. LZSS v0.01 is here!
    By encode in forum Data Compression
    Replies: 67
    Last Post: 28th March 2012, 11:10
  2. Replies: 23
    Last Post: 17th September 2011, 13:12
  3. Google released Snappy compression/decompression library
    By Sportman in forum Data Compression
    Replies: 11
    Last Post: 16th May 2011, 13:31
  4. LZSS with a large dictionary
    By encode in forum Data Compression
    Replies: 31
    Last Post: 31st July 2008, 22:15
  5. Fastest Compressors
    By LovePimple in forum Forum Archive
    Replies: 0
    Last Post: 1st November 2006, 06:36

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •