GDCC public test-set files

[Sportman]

Input:
400,000,000 bytes - TS40.txt (Test 1 - English texts from Project Gutenberg in UTF-8 )

Output:

219,871,172 bytes,    72.520 sec. -    23.897 sec., paq8pxd_v89_no_avx2 -s0
203,837,965 bytes,     2.183 sec. -     0.436 sec., qpress64 -mL2T1
181,027,275 bytes,     4.597 sec. -     0.266 sec., lz4 -3
175,062,733 bytes,     5.991 sec. -     2.596 sec., gzip -1
170,838,185 bytes,     7.909 sec. -     0.260 sec., lz4 -5
169,631,447 bytes,     1.669 sec. -     1.174 sec., brotli_gc82 -q 0 --large_window=30
167,848,522 bytes,    14.472 sec. -     0.264 sec., lz4 -9
162,532,394 bytes,     2.506 sec. -     1.533 sec., tor64 -3 -b1g
160,369,344 bytes,     1.402 sec. -     0.452 sec., zstd -1 --ultra --single-thread
160,062,666 bytes,     8.325 sec. -     2.396 sec., gzip -3
157,075,906 bytes,     1.976 sec. -     1.094 sec., brotli_gc82 -q 1 --large_window=30
150,264,912 bytes,     3.871 sec. -     0.989 sec., brotli_gc82 -q 2 --large_window=30
150,006,975 bytes,     1.954 sec. -     0.549 sec., zstd -2 --ultra --single-thread
149,463,616 bytes,    11.309 sec. -     2.369 sec., gzip -5
149,140,429 bytes,     5.274 sec. -     1.141 sec., brotli_gc82 -q 3 --large_window=30
145,830,570 bytes,    25.139 sec. -     2.336 sec., gzip -9
142,755,486 bytes,     2.334 sec. -     0.563 sec., zstd -3 --ultra --single-thread
142,041,652 bytes,     5.867 sec. -     1.718 sec., rar -m1 -mt1
140,816,232 bytes,     2.358 sec. -     0.591 sec., zstd -4 --ultra --single-thread
140,359,623 bytes,     3.866 sec. -     0.579 sec., zstd -5 --ultra --single-thread
139,837,540 bytes,     9.021 sec. -    11.980 sec., ppmd -m256 -o2
136,868,789 bytes,     5.491 sec. -     0.575 sec., zstd -6 --ultra --single-thread
136,633,636 bytes,     7.333 sec. -     2.353 sec., xwrt -l0
135,555,777 bytes,     6.968 sec. -     1.023 sec., brotli_gc82 -q 4 --large_window=30
134,373,286 bytes,    18.111 sec. -     2.980 sec., 7z -mx1 -t7z -mmt1
132,248,557 bytes,     8.055 sec. -     0.544 sec., zstd -7 --ultra --single-thread
131,148,975 bytes,    12.726 sec. -     1.009 sec., brotli_gc82 -q 5 --large_window=30
130,848,754 bytes,    28.274 sec. -     2.763 sec., 7z -mx3 -t7z -mmt1
130,590,357 bytes,    10.530 sec. -     0.528 sec., zstd -8 --ultra --single-thread
128,877,316 bytes,    16.120 sec. -     0.525 sec., zstd -9 --ultra --single-thread
128,121,552 bytes,    16.978 sec. -     0.994 sec., brotli_gc82 -q 6 --large_window=30
127,948,684 bytes,    18.931 sec. -     0.523 sec., zstd -10 --ultra --single-thread
127,337,551 bytes,    22.643 sec. -     0.528 sec., zstd -11 --ultra --single-thread
125,834,430 bytes,    37.601 sec. -     0.527 sec., zstd -12 --ultra --single-thread
125,827,898 bytes,    30.065 sec. -     4.575 sec., 7z -mx1 -tbzip2 -mmt1
125,566,434 bytes,    26.913 sec. -     0.982 sec., brotli_gc82 -q 7 --large_window=30
125,310,173 bytes,    27.459 sec. -     1.604 sec., rar -m2 -mt1
124,999,622 bytes,     6.261 sec. -     1.007 sec., lzturbo -32 -p1 -b1024
123,308,097 bytes,    45.064 sec. -     0.998 sec., brotli_gc82 -q 8 --large_window=30
123,027,949 bytes,    44.446 sec. -     0.521 sec., zstd -13 --ultra --single-thread
121,198,039 bytes,    58.888 sec. -     0.529 sec., zstd -14 --ultra --single-thread
121,158,907 bytes,    86.535 sec. -     1.034 sec., brotli_gc82 -q 9 --large_window=30
120,950,376 bytes,    67.583 sec. -     1.557 sec., rar -m3 -mt1
119,679,537 bytes,    80.359 sec. -     0.540 sec., zstd -15 --ultra --single-thread
118,232,294 bytes,   149.312 sec. -     1.559 sec., rar -m4 -mt1
117,068,801 bytes,   220.511 sec. -     1.575 sec., rar -m5 -mt1
114,443,789 bytes,    74.329 sec. -     0.515 sec., zstd -16 --ultra --single-thread
113,635,552 bytes,    10.122 sec. -    11.798 sec., ppmd -m256 -o3
112,623,113 bytes,   106.130 sec. -     0.512 sec., zstd -17 --ultra --single-thread
110,930,478 bytes,   125.705 sec. -     0.544 sec., zstd -18 --ultra --single-thread
109,536,996 bytes,   142.226 sec. -     0.542 sec., zstd -19 --ultra --single-thread
109,490,803 bytes,   339.655 sec. -     6.143 sec., 7z -mx9 -tbzip2 -mmt1
106,784,025 bytes,   174.501 sec. -     0.563 sec., zstd -20 --ultra --single-thread
106,763,971 bytes,   179.277 sec. -     2.294 sec., 7z -mx5 -t7z -mmt1
105,010,520 bytes,   197.138 sec. -     0.582 sec., zstd -21 --ultra --single-thread
104,828,445 bytes,   204.698 sec. -     2.369 sec., 7z -mx7 -t7z -mmt1
104,105,584 bytes,   399.093 sec. -     1.321 sec., brotli_gc82 -q 10 --large_window=30
103,573,154 bytes,   250.832 sec. -     1.836 sec., tor64 -16 -b1g
103,449,693 bytes,   216.918 sec. -     0.598 sec., zstd -22 --ultra --single-thread
103,161,326 bytes,   228.660 sec. -     2.444 sec., 7z -mx9 -t7z -mmt1
101,516,046 bytes,   687.137 sec. -     1.256 sec., brotli_gc82 -q 11 --large_window=30
 99,417,161 bytes,    12.430 sec. -    13.326 sec., ppmd -m256 -o4
 91,358,461 bytes,    54.995 sec. -    60.793 sec., ccm 7
 90,370,482 bytes,   799.376 sec. -     4.195 sec., rz -d 1023M
 88,957,140 bytes,    23.784 sec. -    24.095 sec., ppmd -m256 -o6
 87,726,581 bytes,    39.753 sec. -    40.490 sec., ppmd -m256 -o8
 86,739,921 bytes,    56.693 sec. -    50.542 sec., arc -m9 -mt1
 86,549,699 bytes,    30.431 sec. -    13.516 sec., m99_gcc82_k8 -b1000000000 -t1
 84,962,580 bytes,   122.975 sec. -   129.632 sec., cmm4 77
 84,625,414 bytes,   172.474 sec. -     4.419 sec., glza -t1
 83,282,332 bytes,   229.907 sec. -   244.270 sec., lpq1
 83,265,089 bytes,    32.076 sec. -     6.381 sec., bwtturbo -59 -t0
 83,137,161 bytes,    52.545 sec. -    38.736 sec., ringsx64 -m8 -t1
 82,615,027 bytes,   341.316 sec. -   348.223 sec., paq9a -9
 81,773,328 bytes,  2343.278 sec. -  2341.492 sec., uda 0.301
 81,729,538 bytes,    42.922 sec. -    24.991 sec., bsc -m0 -b1024 -e2 -T
 81,681,030 bytes,   222.863 sec. -   245.341 sec., lpaq8 9
 81,233,486 bytes,    43.788 sec. -    42.087 sec., bcm -9
 81,046,263 bytes,   362.984 sec. -   299.654 sec., bbb cm1000
 80,984,961 bytes,    86.155 sec. -    88.385 sec., zcmx64 -m8 -t1
 80,569,729 bytes,   277.914 sec. -   282.126 sec., ppmonstr -m1024
 79,911,160 bytes,   558.554 sec. -   530.958 sec., slim23d -o32 -m2048
 79,270,565 bytes,  1857.265 sec. -  1898.469 sec., fp8_v3 -8
 79,110,045 bytes,   674.670 sec. -   685.552 sec., zpaq64 -m511 -t1
 77,503,484 bytes,   194.256 sec. -   193.304 sec., nz -cc -t1
 76,156,565 bytes,    90.437 sec. -    85.750 sec., mcm_generic -x -m11
 70,648,226 bytes, 24684.416 sec. - 25236.934 sec., paq8pxd_v89_no_avx2 -x15

Input:
400,000,000 bytes - mixed40.dat (Test 3 - Ubuntu x64 distribution and x64 shared library files from Python packages for Linux)

Output:

151,828,132 bytes,     0.550 sec.-      0.261 sec., lz4 -1
128,037,938 bytes,     2.518 sec.-      0.248 sec., lz4 -3
125,575,606 bytes,     3.474 sec.-      0.245 sec., lz4 -5
124,236,658 bytes,     5.143 sec.-      0.241 sec., lz4 -7
123,667,882 bytes,     9.004 sec.-      0.241 sec., lz4 -9
120,343,782 bytes,     0.937 sec.-      0.984 sec., brotli_gc82 -q 0 --large_window=30
112,535,313 bytes,     4.319 sec.-      1.856 sec., gzip -1
109,862,965 bytes,     1.145 sec.-      0.947 sec., brotli_gc82 -q 1 --large_window=30
107,968,743 bytes,     5.423 sec.-      1.801 sec., gzip -3
101,890,860 bytes,     0.869 sec.-      0.429 sec., zstd -1 --ultra --single-thread
100,621,029 bytes,     7.179 sec.-      1.747 sec., gzip -5
 99,121,232 bytes,     1.054 sec.-      0.442 sec., zstd -2 --ultra --single-thread
 98,469,169 bytes,    14.139 sec.-      1.717 sec., gzip -7
 97,624,475 bytes,    86.727 sec.-      1.709 sec., gzip -9
 96,165,328 bytes,     1.487 sec.-      0.458 sec., zstd -4 --ultra --single-thread
 96,114,162 bytes,     1.354 sec.-      0.448 sec., zstd -3 --ultra --single-thread
 94,901,663 bytes,     2.403 sec.-      0.780 sec., brotli_gc82 -q 2 --large_window=30
 93,509,021 bytes,     2.237 sec.-      0.462 sec., zstd -5 --ultra --single-thread
 93,472,235 bytes,    27.393 sec.-      3.190 sec., 7z -mx1 -tbzip2 -mmt1
 93,264,123 bytes,     4.745 sec.-      0.923 sec., brotli_gc82 -q 4 --large_window=30
 93,139,762 bytes,     3.243 sec.-      0.955 sec., brotli_gc82 -q 3 --large_window=30
 92,862,470 bytes,     2.559 sec.-      0.463 sec., zstd -6 --ultra --single-thread
 92,859,347 bytes,    30.555 sec.-      3.870 sec., 7z -mx3 -tbzip2 -mmt1
 92,810,578 bytes,    31.959 sec.-      4.070 sec., 7z -mx5 -tbzip2 -mmt1
 91,930,200 bytes,    79.564 sec.-      3.993 sec., 7z -mx7 -tbzip2 -mmt1
 89,292,399 bytes,   272.160 sec.-      3.558 sec., 7z -mx9 -tbzip2 -mmt1
 88,245,920 bytes,     4.538 sec. -     0.827 sec., lzturbo -32 -b1024 -p1
 87,866,645 bytes,     3.886 sec.-      0.433 sec., zstd -7 --ultra --single-thread
 86,374,802 bytes,     4.649 sec.-      0.423 sec., zstd -8 --ultra --single-thread
 85,881,714 bytes,     5.487 sec.-      0.422 sec., zstd -9 --ultra --single-thread
 85,060,590 bytes,    34.308 sec. -    35.908 sec., ppmd e -m256 -o8
 84,861,877 bytes,     6.438 sec.-      0.421 sec., zstd -10 --ultra --single-thread
 84,634,980 bytes,     7.789 sec.-      0.421 sec., zstd -11 --ultra --single-thread
 84,418,783 bytes,    17.501 sec.-      0.413 sec., zstd -13 --ultra --single-thread
 84,384,933 bytes,    10.286 sec.-      0.418 sec., zstd -12 --ultra --single-thread
 84,246,538 bytes,     5.558 sec.-      1.210 sec., rar -m1 -mt1
 84,171,820 bytes,    19.503 sec.-      0.414 sec., zstd -14 --ultra --single-thread
 84,043,256 bytes,    26.311 sec.-      0.415 sec., zstd -15 --ultra --single-thread
 80,871,756 bytes,    41.564 sec.-      0.427 sec., zstd -16 --ultra --single-thread
 79,062,917 bytes,     8.893 sec.-      0.952 sec., brotli_gc82 -q 5 --large_window=30
 78,796,316 bytes,    20.140 sec. -     6.911 sec., bwtturbo -59 -t0
 78,411,587 bytes,    11.515 sec.-      0.944 sec., brotli_gc82 -q 6 --large_window=30
 78,278,819 bytes,    49.844 sec.-      0.430 sec., zstd -17 --ultra --single-thread
 78,157,770 bytes,   130.282 sec. -   150.564 sec., nz -cc -t1
 77,638,400 bytes,    18.573 sec.-      0.936 sec., brotli_gc82 -q 7 --large_window=30
 77,383,789 bytes,    33.179 sec.-      0.933 sec., brotli_gc82 -q 8 --large_window=30
 76,807,130 bytes,    28.521 sec. -    34.126 sec., bcm -9
 76,631,786 bytes,    73.061 sec.-      0.932 sec., brotli_gc82 -q 9 --large_window=30
 75,951,871 bytes,   446.226 sec. -   348.386 sec., bbb cm1000
 75,824,086 bytes,    25.195 sec. -    20.394 sec., bsc -m0 -b1024 -e2 -T
 75,582,037 bytes,    69.137 sec.-      0.453 sec., zstd -18 --ultra --single-thread
 75,073,993 bytes,    86.311 sec.-      0.454 sec., zstd -19 --ultra --single-thread
 74,969,276 bytes,     9.114 sec.-      2.189 sec., 7z -mx1 -t7z -mmt1
 73,290,454 bytes,    13.004 sec.-      2.070 sec., 7z -mx3 -t7z -mmt1
 72,543,276 bytes,   247.662 sec. -     4.239 sec., glza -t1
 72,486,000 bytes,    93.220 sec.-      0.481 sec., zstd -20 --ultra --single-thread
 70,969,356 bytes,   103.202 sec.-      0.490 sec., zstd -21 --ultra --single-thread
 70,048,687 bytes,     8.352 sec.-      1.167 sec., rar -m2 -mt1
 69,480,082 bytes,   117.975 sec.-      0.501 sec., zstd -22 --ultra --single-thread
 67,674,309 bytes,     9.390 sec.-      1.136 sec., rar -m3 -mt1
 67,362,088 bytes,    13.057 sec.-      1.127 sec., rar -m4 -mt1
 67,243,590 bytes,    14.647 sec.-      1.122 sec., rar -m5 -mt1
 66,556,038 bytes,   450.634 sec.-      1.047 sec., brotli_gc82 -q 10 --large_window=30
 64,798,846 bytes,    80.630 sec.-      1.964 sec., 7z -mx5 -t7z -mmt1
 62,359,340 bytes,   100.765 sec.-      1.928 sec., 7z -mx7 -t7z -mmt1
 62,098,925 bytes,   690.067 sec.-      0.949 sec., brotli_gc82 -q 11 --large_window=30
 61,184,161 bytes,   107.507 sec.-      1.912 sec., 7z -mx9 -t7z -mmt1
 57,126,873 bytes,    56.055 sec. -    54.751 sec., zcmx64 -m8 -t1   
 53,090,885 bytes,    77.583 sec. -   xxx.xxx sec., mcm_generic -x -m11 (decomp take to long)
 52,847,099 bytes,   432.018 sec. -     2.855 sec., rz -d 1023M
 52,546,609 bytes,   560.550 sec. -   xxx.xxx sec., zpaq64 -m511 -t1
 47,100,239 bytes,  1707.788 sec. -  xxxx.xxx sec., fp8_v3 -8

Input:
399,998,976 bytes - block40.dat (Test 4 - roughly 30-70 mixture of Test 1 and Test 3 data)

Output:

184,854,213 bytes,     0.650 sec.-      0.263 sec., lz4 -1
155,565,151 bytes,     3.138 sec.-      0.258 sec., lz4 -3
152,356,335 bytes,     4.384 sec.-      0.255 sec., lz4 -5
151,191,133 bytes,     5.903 sec.-      0.254 sec., lz4 -7
150,812,547 bytes,     8.557 sec.-      0.254 sec., lz4 -9
149,491,434 bytes,     1.217 sec.-      1.137 sec., brotli_gc82 -q 0 --large_window=30
139,468,782 bytes,     4.955 sec.-      2.129 sec., gzip -1
138,482,759 bytes,     1.462 sec.-      1.078 sec., brotli_gc82 -q 1 --large_window=30
136,580,784 bytes,     1.065 sec.-      0.464 sec., zstd -1 --ultra --single-thread
132,017,337 bytes,     6.185 sec.-      2.031 sec., gzip -3
129,446,737 bytes,     1.359 sec.-      0.484 sec., zstd -2 --ultra --single-thread
124,383,697 bytes,     8.262 sec.-      2.005 sec., gzip -5
124,138,139 bytes,     1.766 sec.-      0.492 sec., zstd -3 --ultra --single-thread
122,906,961 bytes,     1.884 sec.-      0.518 sec., zstd -4 --ultra --single-thread
122,531,497 bytes,    14.341 sec.-      1.975 sec., gzip -7
121,946,034 bytes,    60.951 sec.-      1.964 sec., gzip -9
121,728,460 bytes,     3.038 sec.-      0.924 sec., brotli_gc82 -q 2 --large_window=30
120,421,936 bytes,     2.859 sec.-      0.522 sec., zstd -5 --ultra --single-thread
120,133,711 bytes,     4.091 sec.-      1.097 sec., brotli_gc82 -q 3 --large_window=30
119,142,375 bytes,     3.496 sec.-      0.520 sec., zstd -6 --ultra --single-thread
117,361,115 bytes,     5.757 sec.-      1.034 sec., brotli_gc82 -q 4 --large_window=30
114,276,021 bytes,     5.186 sec.-      0.489 sec., zstd -7 --ultra --single-thread
113,460,257 bytes,    27.352 sec.-      3.383 sec., 7z -mx1 -tbzip2 -mmt1
113,254,361 bytes,     6.420 sec.-      0.478 sec., zstd -8 --ultra --single-thread
112,657,116 bytes,     7.839 sec.-      0.477 sec., zstd -9 --ultra --single-thread
112,566,886 bytes,    29.905 sec.-      4.169 sec., 7z -mx3 -tbzip2 -mmt1
111,819,082 bytes,    31.328 sec.-      4.435 sec., 7z -mx5 -tbzip2 -mmt1
111,727,780 bytes,    83.494 sec.-      4.366 sec., 7z -mx7 -tbzip2 -mmt1
111,076,866 bytes,     9.564 sec.-      0.479 sec., zstd -10 --ultra --single-thread
110,377,975 bytes,    12.005 sec.-      0.481 sec., zstd -11 --ultra --single-thread
110,174,286 bytes,   291.329 sec.-      3.867 sec., 7z -mx9 -tbzip2 -mmt1
110,096,027 bytes,     5.556 sec. -     0.933 sec., lzturbo -32 -b1024 -p1
109,989,855 bytes,    16.457 sec.-      0.479 sec., zstd -12 --ultra --single-thread
109,559,003 bytes,    22.140 sec.-      0.473 sec., zstd -13 --ultra --single-thread
108,417,157 bytes,    26.884 sec.-      0.479 sec., zstd -14 --ultra --single-thread
107,712,708 bytes,    35.516 sec.-      0.483 sec., zstd -15 --ultra --single-thread
105,665,543 bytes,    10.864 sec.-      1.093 sec., brotli_gc82 -q 5 --large_window=30
104,533,189 bytes,   153.726 sec. -   125.871 sec., nz -cc -t1
104,259,000 bytes,    14.040 sec.-      1.074 sec., brotli_gc82 -q 6 --large_window=30
103,815,542 bytes,    45.487 sec.-      0.486 sec., zstd -16 --ultra --single-thread
102,667,031 bytes,    22.259 sec.-      1.059 sec., brotli_gc82 -q 7 --large_window=30
101,571,387 bytes,    11.471 sec.-      2.680 sec., 7z -mx1 -t7z -mmt1
101,139,011 bytes,    38.302 sec.-      1.077 sec., brotli_gc82 -q 8 --large_window=30
100,944,178 bytes,    60.172 sec.-      0.488 sec., zstd -17 --ultra --single-thread
 99,411,048 bytes,    82.571 sec.-      1.058 sec., brotli_gc82 -q 9 --large_window=30
 98,638,685 bytes,    78.266 sec.-      0.511 sec., zstd -18 --ultra --single-thread
 98,385,901 bytes,    18.605 sec.-      2.450 sec., 7z -mx3 -t7z -mmt1
 97,641,046 bytes,    93.578 sec.-      0.521 sec., zstd -19 --ultra --single-thread
 93,821,193 bytes,   109.980 sec.-      0.560 sec., zstd -20 --ultra --single-thread
 93,575,920 bytes,    40.051 sec. -    41.667 sec., ppmd -m256 -o8
 91,460,782 bytes,   124.530 sec.-      0.568 sec., zstd -21 --ultra --single-thread
 89,314,971 bytes,   140.094 sec.-      0.581 sec., zstd -22 --ultra --single-thread
 85,534,704 bytes,   104.134 sec.-      2.217 sec., 7z -mx5 -t7z -mmt1
 84,246,538 bytes,     5.558 sec.-      1.210 sec., rar -m1 -mt1
 83,980,924 bytes,   477.131 sec.-      1.223 sec., brotli_gc82 -q 10 --large_window=30
 82,928,976 bytes,    23.939 sec. -     7.xxx sec., bwtturbo -59 -t0 (decomp crash)
 82,861,609 bytes,   125.527 sec.-      2.203 sec., 7z -mx7 -t7z -mmt1
 80,927,550 bytes,   137.540 sec.-      2.209 sec., 7z -mx9 -t7z -mmt1
 80,635,084 bytes,    32.600 sec. -    37.356 sec., bcm -9
 80,511,030 bytes,   222.754 sec. -     6.201 sec., glza -t1
 80,249,048 bytes,    25.195 sec. -    20.394 sec., bsc -m0 -b1024 -e2 -T
 80,118,058 bytes,   372.074 sec. -   355.784 sec., bbb cm1000
 79,968,657 bytes,   734.674 sec.-      1.147 sec., brotli_gc82 -q 11 --large_window=30
 70,971,803 bytes,    68.454 sec. -    67.543 sec., zcmx64 -m8 -t1
 70,048,687 bytes,     8.352 sec.-      1.167 sec., rar -m2 -mt1
 69,992,690 bytes,   510.133 sec. -     3.691 sec., rz -d 1023M
 67,674,309 bytes,     9.390 sec.-      1.136 sec., rar -m3 -mt1
 67,362,088 bytes,    13.057 sec.-      1.127 sec., rar -m4 -mt1
 67,243,590 bytes,    14.647 sec.-      1.122 sec., rar -m5 -mt1
 65,152,519 bytes,   582.133 sec. -   xxx.xxx sec., zpaq64 -m511 -t1
 64,214,420 bytes,    85.848 sec. -   xxx.xxx sec., mcm_generic -x -m11 (decomp take to long)
 61,678,970 bytes,  1787.655 sec. -  xxxx.xxx sec., fp8_v3 -8

[suryakandau@yahoo.co.id]

81,046,263 bytes 1360.32 sec - 1175.06 sec bbbv1.9 cm1000

[Sportman]

81,046,263 bytes 1360.32 sec - 1175.06 sec bbbv1.9 cm1000

Added.

[lz77]

I'm interested in the time of compression + decompression on the I7 8700 CPU of the entire file. Let Sportman's computer 2 times faster than I7 8700. In this case on the Sportman's super computer time of compression + time of decompression must be smaller than 40 sec./2/2.5=8 sec. to meet the rapid compression test conditions.

I do not see the result for lzturbo. Maybe it will be the best.

[Sportman]

Let Sportman's computer 2 times faster than I7 8700.

i7-8700K:
Max Turbo Frequency 4.70 GHz
Intel Smart Cache 12 MB
Memory Types DDR4-2666 (used 2400 MHz)

i9-9900K:
Max Turbo Frequency 5.00 GHz (used OC to 5.20 GHz)
Intel Smart Cache 16 MB
Memory Types DDR4-2666 (used 4266 MHz)

My guess is 11.3% faster for CPU intensive tasks.

I do not see the result for lzturbo.

bwtturbo -59 was tested.

[suryakandau@yahoo.co.id]

i7-8700K:
Max Turbo Frequency 4.70 GHz
Intel Smart Cache 12 MB
Memory Types DDR4-2666 (used 2400 MHz)

i9-9900K:
Max Turbo Frequency 5.00 GHz (used OC to 5.20 GHz)
Intel Smart Cache 16 MB
Memory Types DDR4-2666 (used 4266 MHz)

My guess is 11.3% faster for CPU intensive tasks.



bwtturbo -59 was tested.

I do not see the result for paq8sk23 -x15. Maybe it will be the best.

[Sportman]

I do not see the result for paq8sk23 -x15.

I'm afraid you must make it first at least 36 times quicker...

Compression + Decompression Time Limit, seconds:
T1 High compression ratio, < 4000
T2 High compression ratio, < 4000
T3 High compression ratio, < 4000
T4 High compression ratio, < 4000

[suryakandau@yahoo.co.id]

I'm afraid you must make it first at least 36 times quicker...

Compression + Decompression Time Limit, seconds:
T1 High compression ratio, < 4000
T2 High compression ratio, < 4000
T3 High compression ratio, < 4000
T4 High compression ratio, < 4000

Maybe you can test paq8sk19. It is the fastest version of paq8sk series

[suryakandau@yahoo.co.id]

i have fixed a little in word model paq8pxd_89. @sportman could you test it using -x15 ? maybe it can compress better than the original

[Shelwien]

All paq8 versions are too slow for the competition - even fp8 (since TS40 is 40% of whole file and there's a 4000s time limit).

[suryakandau@yahoo.co.id]

All paq8 versions are too slow for the competition - even fp8 (since TS40 is 40% of whole file and there's a 4000s time limit).

So what algorithm suitable for this competition ?

[lz77]

So what algorithm suitable for this competition ?

I think, compressors will have a size of 40MB and will contain a part of the file TS40.txt in a very compressed form. Or something like that.

[paleski]

Input:
400,000,000 bytes - TS40.txt (English texts from Project Gutenberg in UTF-8 )

Output:

How about UDA, maybe it also will compress within a time limit?

[Sportman]

How about UDA

Added.

[Sportman]

My guess is 11.3% faster for CPU intensive tasks.

I tested PPMII variant J, May 20 2006:
39.753 sec. - 40.490 sec., ppmd -m256 -o8 (default -r0)
130.028 sec. - 132.782 sec., ppmd -m256 -o8 -r0 (https://globalcompetition.compression.ru/rules/)
52.011 sec. - 53.113 sec. (divided by 1000MB/400MB=2.5)
1.31x slower - 1.31x slower

At 800 euro micro ATX desktop PC i7-8559U (4 core x 2.7GHz 4.5GHz turbo max, 2 x 8GB memory 2400MHz 14-14-14-35, 1TB NVMe 3500MB/s read, 3300MB/s write):
51.427 sec. - 52.013 sec., ppmd -m256 -o8 (default -r0)
1.29x slower - 1.28x slower

Both systems use Windows 10 Pro, Power options at High performance, only needed services enabled and Meltdown and Spectre patches disabled.

[lz77]

> 99,417,161 bytes, 12.430 sec. - 13.326 sec., ppmd -m256 -o4

Is it (de)compressed in single thread??

[Sportman]

>ppmd -m256 -o4
Is it (de)compressed in single thread??

Yes, both operations are single thread.

[lz77]

Oh, I see, a PPM algorithm can't be winner in Rapid compression:


> 113,635,552 bytes, 10.122 sec. - 11.798 sec., ppmd -m256 -o3

(10.122+11.79) *2.5*1.113 > 40 sec.


Is zstd (LZ77+rANS?) to be winner in the list above?

> 130,590,357 bytes, 10.530 sec. - 0.528 sec., zstd -8 --ultra --single-thread

[Shelwien]

> 113,635,552 bytes, 10.122 sec. - 11.798 sec., ppmd -m256 -o3
> (10.122+11.79) *2.5*1.113 > 40 sec.

Sure, but its possible (basically intended) to use preprocessing before compression,
which would reduce 1Gb of text to around 600Mb (in case of WRT).

> Is zstd (LZ77+rANS?) to be winner in the list above?

zstd is LZ77+huffman mostly. It kinda uses tANS for matches, but FSE tANS isn't that much better than huffman.

Also zstd encoding is slower than PPM at high levels, and isn't very good for text compression.

[lz77]

But ppmd in this list did not used text preprocessing??

[Shelwien]

> But ppmd in this list did not used text preprocessing??

Its not, though Sportman could process TS40 with paq8pxd -s0 or something.

I'm just saying that contest ranking of codecs is not that clear.

[Sportman]

though Sportman could process TS40 with paq8pxd -s0 or something.

Added.

[Shelwien]

paq8pxd -s0 is a preprocessor, I meant that it can be combined with ppmd and some other codecs that lack integrated preprocessing (eg. mcm has it).

[lz77]

paq8pxd_v89_no_avx2 -s0 works 72 sec. Too slow preprocessing...
I'm interested in the file size after maximum Rapid compression (with a condition of 40/2.5 sec. for compression + decompression).

[Shelwien]

> paq8pxd_v89_no_avx2 -s0 works 72 sec. Too slow preprocessing...

Decoding is 24s though. Also there're others: xwrt, DRT, Shkarin's "liptify", Bulat's dict, cmix -s, mcm.
Even WRT/LIPT is a simple algorithm, which has no reason to be slow (aside from inefficient implementation).

> I'm interested in the file size after maximum Rapid compression (with a condition of 40/2.5 sec. for compression + decompression).

Maybe try tweaking zstd? Like https://github.com/facebook/zstd/blo...s/paramgrill.c ?

[lz77]

I think, lzturbo by DND after tuning has a good chance, but B. Ziganshin has a claim to this program... Has lzturbo the right to participate in the Competition?

[suryakandau@yahoo.co.id]

81,046,263 bytes 1360.32 sec - 1175.06 sec bbbv1.9 cm1000

bbb v1.10
400000000 -> 81046263 in 1071.57 sec
faster than v1.9

[Darek]

paq8sk30 scores:

70'587'620 - TS40.txt -x15 by Paq8sk30, time - 38'154,53s
70'315'001 - TS40.txt -x15 -e1,english.dic by Paq8sk30, time - 45'253,18s
72'656'037 - TS40.txt -x15 -w -e1,english.dic by Paq8sk30, time - 89'471,51s - worse score, bad time, I don't made decompression test...

[suryakandau@yahoo.co.id]

paq8sk30 scores:

70'587'620 - TS40.txt -x15 by Paq8sk30, time - 38'154,53s
70'315'001 - TS40.txt -x15 -e1,english.dic by Paq8sk30, time - 45'253,18s
72'656'037 - TS40.txt -x15 -w -e1,english.dic by Paq8sk30, time - 89'471,51s - worse score, bad time, I don't made decompression test...

so the best score is using -x15 -e1,english.dic. @sportman could you add it to GDCC public test set file ?

[suryakandau@yahoo.co.id]

paq8sk30 scores:

70'587'620 - TS40.txt -x15 by Paq8sk30, time - 38'154,53s
70'315'001 - TS40.txt -x15 -e1,english.dic by Paq8sk30, time - 45'253,18s
72'656'037 - TS40.txt -x15 -w -e1,english.dic by Paq8sk30, time - 89'471,51s - worse score, bad time, I don't made decompression test...

paq8pxd_v89 when using -w option cause error message
Transform fails at 333440671, skipping...
so it detect ts40.txt as default , not bigtext wrt then that cause the compression ratio is worse