large window brotli results available for a several corpora

[Jyrki Alakuijala]

Aggregate results over all corpora on 'compressionette' -- https://github.com/google/brotli/issues/642

Algorithm + settings     Bytes       Relative to lzma 9d29:fb273:mf=bt4

-----------------------------------------------------------------------

lzma 9d29:fb273:mf=bt4   836785740   +- 0 %

brotli 11d29             842290211   + 0.6 %

lzham 4fb258:x4:d29      845777637   + 1 %

lzturbo 39               852928550   + 2 %

zstd 22                  880696654   + 5 %

oodle 129                888517690   + 6 %

[Sportman]

Input:
4,399,079,936 bytes - Superpig_Corpus.tar

Output:
  688,477,808 bytes,  7700.042 sec., 7807.149 sec., Zpaq - 510
  707,833,045 bytes,   799.819 sec.,  783.293 sec., ZCM - 8
  709,321,836 bytes,  3162.060 sec., 3112.839 sec., NanoZip - c
  720,849,403 bytes,  1095.231 sec., 1065.008 sec., MCM - x
  741,250,474 bytes,  7515.394 sec.,   33.856 sec., RAZOR - -d 1023M
  755,547,809 bytes,  6215.010 sec.,   32.202 sec., RAZOR - default
  770,048,219 bytes,   885.474 sec.,  220.076 sec., NanoZip - O
  842,519,409 bytes,  2189.817 sec.,   38.406 sec., FreeArc - 9
  843,555,914 bytes,  7794.198 sec.,   10.805 sec., Brotli - 11 --large_window=30
  846,879,758 bytes, 22757.219 sec.,   13.712 sec., LZHAM - -m4 -d29 -x8 -f257 -b
  864,728,456 bytes,   158.403 sec.,  241.753 sec., Bsc - 6
  870,176,977 bytes,  2104.634 sec.,   22.709 sec., 7-Zip - 9
  885,675,409 bytes,  2671.613 sec.,    6.167 sec., Zstd - 22
  894,437,507 bytes,   329.482 sec.,   42.154 sec., Lrzip - 9
  913,130,154 bytes,  6975.268 sec.,    9.422 sec., Brotli - 11
  966,743,794 bytes,  1404.794 sec.,   13.755 sec., WinRAR - 5
1,033,164,470 bytes,  3618.509 sec.,   56.856 sec., Bzip2 - 9
1,250,969,770 bytes,  2657.982 sec.,   15.766 sec., Zip - 9

[Jyrki Alakuijala]

Which window size did you use with brotli?

[Sportman]

Which window size did you use with brotli?

Default, I tried at an other big file to compress with -w 24 parameter but got same output file size and speeds as without that parameter, so I assumed Brotli automatic choose optimal value.

[JamesB]

Jyrki's test above has "11d29" in the name. Although -d isn't selecting the size, this is presumably to match the 7zip parameter in name and *would* be equivalent to -w 29, if that was possible in the current release.

The brotli/c/tools/brotli.c code appears to support large window sizes and has this:

      /* Specified by user. */
      /* Do not enable "large-window" extension, if not required. */
      if (context->lgwin > BROTLI_MAX_WINDOW_BITS) {
        BrotliEncoderSetParameter(s, BROTLI_PARAM_LARGE_WINDOW, 1u);
      }
      BrotliEncoderSetParameter(s,
          BROTLI_PARAM_LGWIN, (uint32_t)context->lgwin);

So it enables large window mode if we specify window of 25 or more, but then it's rejected somewhere else. Any clues how to enable the large window mode?

Edit - this patch blocks the warning and it's running now. Will see what it does.
*** DOESN'T HELP ***. It works and can be decoded, but -w29 output is now larger than -w24.

diff --git a/c/tools/brotli.c b/c/tools/brotli.c
index ea4fdac..f85b073 100644
--- a/c/tools/brotli.c
+++ b/c/tools/brotli.c
@@ -335,7 +335,7 @@ static Command ParseParams(Context* params) {
             return COMMAND_INVALID;
           }
           lgwin_set = ParseInt(argv[i], 0,
-                               BROTLI_MAX_WINDOW_BITS, &params->lgwin);
+                               BROTLI_LARGE_MAX_WINDOW_BITS, &params->lgwin);
           if (!lgwin_set) {
             fprintf(stderr, "error parsing lgwin value [%s]\n", argv[i]);
             return COMMAND_INVALID;

[Jyrki Alakuijala]

Default, I tried at an other big file to compress with -w 24 parameter but got same output file size and speeds as without that parameter, so I assumed Brotli automatic choose optimal value.

Brotli's basic mode restricts backward reference window to 16 MB. It is not useful to compare 16 MB backward reference window to 32 MB or more with another codec. It is not the same use case.

This is how you invoke the large window mode (only with -q 11 for now):

$ brotli -q 11 --large_window=30 --output=a.br bigfile.dat

[Jyrki Alakuijala]

If it is confusing to use, please file a bug on github.com/google/brotli

[Sportman]

This is how you invoke the large window mode (only with -q 11 for now):

$ brotli -q 11 --large_window=30 --output=a.br bigfile.dat

Thanks, with this option I get a smaller file with lower speeds, what's the max "large window" value 30?

[Jyrki Alakuijala]

Thanks, with this option I get a smaller file with lower speeds, what's the max "large window" value 30?

30 bits, i.e., 1 GB is the max of the current implementation. https://github.com/google/brotli/blo...on/constants.h

[Sportman]

Added Superpig_Corpus.tar result with --large_window=30.