Results 1 to 10 of 10

Thread: large window brotli results available for a several corpora

  1. #1
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    846
    Thanks
    242
    Thanked 309 Times in 184 Posts

    large window brotli results available for a several corpora

    Aggregate results over all corpora on 'compressionette' -- https://github.com/google/brotli/issues/642


    Algorithm + settings Bytes Relative to lzma 9d29:fb273:mf=bt4
    -----------------------------------------------------------------------
    lzma 9d29:fb273:mf=bt4 836785740 +- 0 %
    brotli 11d29 842290211 + 0.6 %
    lzham 4fb258:x4:d29 845777637 + 1 %
    lzturbo 39 852928550 + 2 %
    zstd 22 880696654 + 5 %
    oodle 129 888517690 + 6 %
    Last edited by Jyrki Alakuijala; 1st May 2018 at 16:17.

  2. Thanks (2):

    khavish (1st May 2018),Sportman (1st May 2018)

  3. #2
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    1,005
    Thanks
    97
    Thanked 399 Times in 278 Posts
    Code:
    Input:
    4,399,079,936 bytes - Superpig_Corpus.tar
    
    Output:
      688,477,808 bytes,  7700.042 sec., 7807.149 sec., Zpaq - 510
      707,833,045 bytes,   799.819 sec.,  783.293 sec., ZCM - 8
      709,321,836 bytes,  3162.060 sec., 3112.839 sec., NanoZip - c
      720,849,403 bytes,  1095.231 sec., 1065.008 sec., MCM - x
      741,250,474 bytes,  7515.394 sec.,   33.856 sec., RAZOR - -d 1023M
      755,547,809 bytes,  6215.010 sec.,   32.202 sec., RAZOR - default
      770,048,219 bytes,   885.474 sec.,  220.076 sec., NanoZip - O
      842,519,409 bytes,  2189.817 sec.,   38.406 sec., FreeArc - 9
      843,555,914 bytes,  7794.198 sec.,   10.805 sec., Brotli - 11 --large_window=30
      846,879,758 bytes, 22757.219 sec.,   13.712 sec., LZHAM - -m4 -d29 -x8 -f257 -b
      864,728,456 bytes,   158.403 sec.,  241.753 sec., Bsc - 6
      870,176,977 bytes,  2104.634 sec.,   22.709 sec., 7-Zip - 9
      885,675,409 bytes,  2671.613 sec.,    6.167 sec., Zstd - 22
      894,437,507 bytes,   329.482 sec.,   42.154 sec., Lrzip - 9
      913,130,154 bytes,  6975.268 sec.,    9.422 sec., Brotli - 11
      966,743,794 bytes,  1404.794 sec.,   13.755 sec., WinRAR - 5
    1,033,164,470 bytes,  3618.509 sec.,   56.856 sec., Bzip2 - 9
    1,250,969,770 bytes,  2657.982 sec.,   15.766 sec., Zip - 9
    Last edited by Sportman; 3rd May 2018 at 22:35. Reason: Added Bzip2, NanoZip, RAZOR, Zpaq, LZHAM

  4. #3
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    846
    Thanks
    242
    Thanked 309 Times in 184 Posts
    Which window size did you use with brotli?
    Last edited by Jyrki Alakuijala; 10th May 2018 at 13:51.

  5. #4
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    1,005
    Thanks
    97
    Thanked 399 Times in 278 Posts
    Quote Originally Posted by Jyrki Alakuijala View Post
    Which window size did you use with brotli?
    Default, I tried at an other big file to compress with -w 24 parameter but got same output file size and speeds as without that parameter, so I assumed Brotli automatic choose optimal value.

  6. #5
    Member
    Join Date
    Dec 2011
    Location
    Cambridge, UK
    Posts
    506
    Thanks
    186
    Thanked 177 Times in 120 Posts
    Jyrki's test above has "11d29" in the name. Although -d isn't selecting the size, this is presumably to match the 7zip parameter in name and *would* be equivalent to -w 29, if that was possible in the current release.

    The brotli/c/tools/brotli.c code appears to support large window sizes and has this:

    Code:
          /* Specified by user. */
          /* Do not enable "large-window" extension, if not required. */
          if (context->lgwin > BROTLI_MAX_WINDOW_BITS) {
            BrotliEncoderSetParameter(s, BROTLI_PARAM_LARGE_WINDOW, 1u);
          }
          BrotliEncoderSetParameter(s,
              BROTLI_PARAM_LGWIN, (uint32_t)context->lgwin);
    So it enables large window mode if we specify window of 25 or more, but then it's rejected somewhere else. Any clues how to enable the large window mode?

    Edit - this patch blocks the warning and it's running now. Will see what it does.
    *** DOESN'T HELP ***. It works and can be decoded, but -w29 output is now larger than -w24.

    Code:
    diff --git a/c/tools/brotli.c b/c/tools/brotli.c
    index ea4fdac..f85b073 100644
    --- a/c/tools/brotli.c
    +++ b/c/tools/brotli.c
    @@ -335,7 +335,7 @@ static Command ParseParams(Context* params) {
                 return COMMAND_INVALID;
               }
               lgwin_set = ParseInt(argv[i], 0,
    -                               BROTLI_MAX_WINDOW_BITS, &params->lgwin);
    +                               BROTLI_LARGE_MAX_WINDOW_BITS, &params->lgwin);
               if (!lgwin_set) {
                 fprintf(stderr, "error parsing lgwin value [%s]\n", argv[i]);
                 return COMMAND_INVALID;

  7. #6
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    846
    Thanks
    242
    Thanked 309 Times in 184 Posts
    Quote Originally Posted by Sportman View Post
    Default, I tried at an other big file to compress with -w 24 parameter but got same output file size and speeds as without that parameter, so I assumed Brotli automatic choose optimal value.
    Brotli's basic mode restricts backward reference window to 16 MB. It is not useful to compare 16 MB backward reference window to 32 MB or more with another codec. It is not the same use case.

    This is how you invoke the large window mode (only with -q 11 for now):

    $ brotli -q 11 --large_window=30 --output=a.br bigfile.dat

  8. Thanks:

    Sportman (3rd May 2018)

  9. #7
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    846
    Thanks
    242
    Thanked 309 Times in 184 Posts
    If it is confusing to use, please file a bug on github.com/google/brotli

  10. #8
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    1,005
    Thanks
    97
    Thanked 399 Times in 278 Posts
    Quote Originally Posted by Jyrki Alakuijala View Post
    This is how you invoke the large window mode (only with -q 11 for now):

    $ brotli -q 11 --large_window=30 --output=a.br bigfile.dat
    Thanks, with this option I get a smaller file with lower speeds, what's the max "large window" value 30?

  11. #9
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    846
    Thanks
    242
    Thanked 309 Times in 184 Posts
    Quote Originally Posted by Sportman View Post
    Thanks, with this option I get a smaller file with lower speeds, what's the max "large window" value 30?
    30 bits, i.e., 1 GB is the max of the current implementation. https://github.com/google/brotli/blo...on/constants.h

  12. Thanks (2):

    hunman (3rd May 2018),Sportman (3rd May 2018)

  13. #10
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    1,005
    Thanks
    97
    Thanked 399 Times in 278 Posts
    Added Superpig_Corpus.tar result with --large_window=30.

  14. Thanks:

    Jyrki Alakuijala (10th May 2018)

Similar Threads

  1. Broken window compression
    By SolidComp in forum Data Compression
    Replies: 1
    Last Post: 10th April 2018, 15:45
  2. Large-window brotli (incompatible with standard brotli)
    By Jyrki Alakuijala in forum Data Compression
    Replies: 0
    Last Post: 4th October 2016, 01:45
  3. Replies: 7
    Last Post: 24th September 2015, 06:03
  4. Replies: 3
    Last Post: 30th July 2011, 14:48
  5. Strange gcc4.3 results with paq8o8
    By Hahobas in forum Forum Archive
    Replies: 8
    Last Post: 22nd March 2008, 19:44

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •