Page 1 of 2 12 LastLast
Results 1 to 30 of 37

Thread: GDCC public test-set files

  1. #1
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    1,003
    Thanks
    97
    Thanked 399 Times in 278 Posts

    GDCC public test-set files

    Input:
    400,000,000 bytes - TS40.txt (Test 1 - English texts from Project Gutenberg in UTF-8 )

    Output:
    Code:
    219,871,172 bytes,    72.520 sec. -    23.897 sec., paq8pxd_v89_no_avx2 -s0
    203,837,965 bytes,     2.183 sec. -     0.436 sec., qpress64 -mL2T1
    181,027,275 bytes,     4.597 sec. -     0.266 sec., lz4 -3
    175,062,733 bytes,     5.991 sec. -     2.596 sec., gzip -1
    170,838,185 bytes,     7.909 sec. -     0.260 sec., lz4 -5
    169,631,447 bytes,     1.669 sec. -     1.174 sec., brotli_gc82 -q 0 --large_window=30
    167,848,522 bytes,    14.472 sec. -     0.264 sec., lz4 -9
    162,532,394 bytes,     2.506 sec. -     1.533 sec., tor64 -3 -b1g
    160,369,344 bytes,     1.402 sec. -     0.452 sec., zstd -1 --ultra --single-thread
    160,062,666 bytes,     8.325 sec. -     2.396 sec., gzip -3
    157,075,906 bytes,     1.976 sec. -     1.094 sec., brotli_gc82 -q 1 --large_window=30
    150,264,912 bytes,     3.871 sec. -     0.989 sec., brotli_gc82 -q 2 --large_window=30
    150,006,975 bytes,     1.954 sec. -     0.549 sec., zstd -2 --ultra --single-thread
    149,463,616 bytes,    11.309 sec. -     2.369 sec., gzip -5
    149,140,429 bytes,     5.274 sec. -     1.141 sec., brotli_gc82 -q 3 --large_window=30
    145,830,570 bytes,    25.139 sec. -     2.336 sec., gzip -9
    142,755,486 bytes,     2.334 sec. -     0.563 sec., zstd -3 --ultra --single-thread
    142,041,652 bytes,     5.867 sec. -     1.718 sec., rar -m1 -mt1
    140,816,232 bytes,     2.358 sec. -     0.591 sec., zstd -4 --ultra --single-thread
    140,359,623 bytes,     3.866 sec. -     0.579 sec., zstd -5 --ultra --single-thread
    139,837,540 bytes,     9.021 sec. -    11.980 sec., ppmd -m256 -o2
    136,868,789 bytes,     5.491 sec. -     0.575 sec., zstd -6 --ultra --single-thread
    136,633,636 bytes,     7.333 sec. -     2.353 sec., xwrt -l0
    135,555,777 bytes,     6.968 sec. -     1.023 sec., brotli_gc82 -q 4 --large_window=30
    134,373,286 bytes,    18.111 sec. -     2.980 sec., 7z -mx1 -t7z -mmt1
    132,248,557 bytes,     8.055 sec. -     0.544 sec., zstd -7 --ultra --single-thread
    131,148,975 bytes,    12.726 sec. -     1.009 sec., brotli_gc82 -q 5 --large_window=30
    130,848,754 bytes,    28.274 sec. -     2.763 sec., 7z -mx3 -t7z -mmt1
    130,590,357 bytes,    10.530 sec. -     0.528 sec., zstd -8 --ultra --single-thread
    128,877,316 bytes,    16.120 sec. -     0.525 sec., zstd -9 --ultra --single-thread
    128,121,552 bytes,    16.978 sec. -     0.994 sec., brotli_gc82 -q 6 --large_window=30
    127,948,684 bytes,    18.931 sec. -     0.523 sec., zstd -10 --ultra --single-thread
    127,337,551 bytes,    22.643 sec. -     0.528 sec., zstd -11 --ultra --single-thread
    125,834,430 bytes,    37.601 sec. -     0.527 sec., zstd -12 --ultra --single-thread
    125,827,898 bytes,    30.065 sec. -     4.575 sec., 7z -mx1 -tbzip2 -mmt1
    125,566,434 bytes,    26.913 sec. -     0.982 sec., brotli_gc82 -q 7 --large_window=30
    125,310,173 bytes,    27.459 sec. -     1.604 sec., rar -m2 -mt1
    124,999,622 bytes,     6.261 sec. -     1.007 sec., lzturbo -32 -p1 -b1024
    123,308,097 bytes,    45.064 sec. -     0.998 sec., brotli_gc82 -q 8 --large_window=30
    123,027,949 bytes,    44.446 sec. -     0.521 sec., zstd -13 --ultra --single-thread
    121,198,039 bytes,    58.888 sec. -     0.529 sec., zstd -14 --ultra --single-thread
    121,158,907 bytes,    86.535 sec. -     1.034 sec., brotli_gc82 -q 9 --large_window=30
    120,950,376 bytes,    67.583 sec. -     1.557 sec., rar -m3 -mt1
    119,679,537 bytes,    80.359 sec. -     0.540 sec., zstd -15 --ultra --single-thread
    118,232,294 bytes,   149.312 sec. -     1.559 sec., rar -m4 -mt1
    117,068,801 bytes,   220.511 sec. -     1.575 sec., rar -m5 -mt1
    114,443,789 bytes,    74.329 sec. -     0.515 sec., zstd -16 --ultra --single-thread
    113,635,552 bytes,    10.122 sec. -    11.798 sec., ppmd -m256 -o3
    112,623,113 bytes,   106.130 sec. -     0.512 sec., zstd -17 --ultra --single-thread
    110,930,478 bytes,   125.705 sec. -     0.544 sec., zstd -18 --ultra --single-thread
    109,536,996 bytes,   142.226 sec. -     0.542 sec., zstd -19 --ultra --single-thread
    109,490,803 bytes,   339.655 sec. -     6.143 sec., 7z -mx9 -tbzip2 -mmt1
    106,784,025 bytes,   174.501 sec. -     0.563 sec., zstd -20 --ultra --single-thread
    106,763,971 bytes,   179.277 sec. -     2.294 sec., 7z -mx5 -t7z -mmt1
    105,010,520 bytes,   197.138 sec. -     0.582 sec., zstd -21 --ultra --single-thread
    104,828,445 bytes,   204.698 sec. -     2.369 sec., 7z -mx7 -t7z -mmt1
    104,105,584 bytes,   399.093 sec. -     1.321 sec., brotli_gc82 -q 10 --large_window=30
    103,573,154 bytes,   250.832 sec. -     1.836 sec., tor64 -16 -b1g
    103,449,693 bytes,   216.918 sec. -     0.598 sec., zstd -22 --ultra --single-thread
    103,161,326 bytes,   228.660 sec. -     2.444 sec., 7z -mx9 -t7z -mmt1
    101,516,046 bytes,   687.137 sec. -     1.256 sec., brotli_gc82 -q 11 --large_window=30
     99,417,161 bytes,    12.430 sec. -    13.326 sec., ppmd -m256 -o4
     91,358,461 bytes,    54.995 sec. -    60.793 sec., ccm 7
     90,370,482 bytes,   799.376 sec. -     4.195 sec., rz -d 1023M
     88,957,140 bytes,    23.784 sec. -    24.095 sec., ppmd -m256 -o6
     87,726,581 bytes,    39.753 sec. -    40.490 sec., ppmd -m256 -o8
     86,739,921 bytes,    56.693 sec. -    50.542 sec., arc -m9 -mt1
     86,549,699 bytes,    30.431 sec. -    13.516 sec., m99_gcc82_k8 -b1000000000 -t1
     84,962,580 bytes,   122.975 sec. -   129.632 sec., cmm4 77
     84,625,414 bytes,   172.474 sec. -     4.419 sec., glza -t1
     83,282,332 bytes,   229.907 sec. -   244.270 sec., lpq1
     83,265,089 bytes,    32.076 sec. -     6.381 sec., bwtturbo -59 -t0
     83,137,161 bytes,    52.545 sec. -    38.736 sec., ringsx64 -m8 -t1
     82,615,027 bytes,   341.316 sec. -   348.223 sec., paq9a -9
     81,773,328 bytes,  2343.278 sec. -  2341.492 sec., uda 0.301
     81,729,538 bytes,    42.922 sec. -    24.991 sec., bsc -m0 -b1024 -e2 -T
     81,681,030 bytes,   222.863 sec. -   245.341 sec., lpaq8 9
     81,233,486 bytes,    43.788 sec. -    42.087 sec., bcm -9
     81,046,263 bytes,   362.984 sec. -   299.654 sec., bbb cm1000
     80,984,961 bytes,    86.155 sec. -    88.385 sec., zcmx64 -m8 -t1
     80,569,729 bytes,   277.914 sec. -   282.126 sec., ppmonstr -m1024
     79,911,160 bytes,   558.554 sec. -   530.958 sec., slim23d -o32 -m2048
     79,270,565 bytes,  1857.265 sec. -  1898.469 sec., fp8_v3 -8
     79,110,045 bytes,   674.670 sec. -   685.552 sec., zpaq64 -m511 -t1
     77,503,484 bytes,   194.256 sec. -   193.304 sec., nz -cc -t1
     76,156,565 bytes,    90.437 sec. -    85.750 sec., mcm_generic -x -m11
     70,648,226 bytes, 24684.416 sec. - 25236.934 sec., paq8pxd_v89_no_avx2 -x15

    Input:
    400,000,000 bytes - mixed40.dat (Test 3 - Ubuntu x64 distribution and x64 shared library files from Python packages for Linux)

    Output:
    Code:
    151,828,132 bytes,     0.550 sec.-      0.261 sec., lz4 -1
    128,037,938 bytes,     2.518 sec.-      0.248 sec., lz4 -3
    125,575,606 bytes,     3.474 sec.-      0.245 sec., lz4 -5
    124,236,658 bytes,     5.143 sec.-      0.241 sec., lz4 -7
    123,667,882 bytes,     9.004 sec.-      0.241 sec., lz4 -9
    120,343,782 bytes,     0.937 sec.-      0.984 sec., brotli_gc82 -q 0 --large_window=30
    112,535,313 bytes,     4.319 sec.-      1.856 sec., gzip -1
    109,862,965 bytes,     1.145 sec.-      0.947 sec., brotli_gc82 -q 1 --large_window=30
    107,968,743 bytes,     5.423 sec.-      1.801 sec., gzip -3
    101,890,860 bytes,     0.869 sec.-      0.429 sec., zstd -1 --ultra --single-thread
    100,621,029 bytes,     7.179 sec.-      1.747 sec., gzip -5
     99,121,232 bytes,     1.054 sec.-      0.442 sec., zstd -2 --ultra --single-thread
     98,469,169 bytes,    14.139 sec.-      1.717 sec., gzip -7
     97,624,475 bytes,    86.727 sec.-      1.709 sec., gzip -9
     96,165,328 bytes,     1.487 sec.-      0.458 sec., zstd -4 --ultra --single-thread
     96,114,162 bytes,     1.354 sec.-      0.448 sec., zstd -3 --ultra --single-thread
     94,901,663 bytes,     2.403 sec.-      0.780 sec., brotli_gc82 -q 2 --large_window=30
     93,509,021 bytes,     2.237 sec.-      0.462 sec., zstd -5 --ultra --single-thread
     93,472,235 bytes,    27.393 sec.-      3.190 sec., 7z -mx1 -tbzip2 -mmt1
     93,264,123 bytes,     4.745 sec.-      0.923 sec., brotli_gc82 -q 4 --large_window=30
     93,139,762 bytes,     3.243 sec.-      0.955 sec., brotli_gc82 -q 3 --large_window=30
     92,862,470 bytes,     2.559 sec.-      0.463 sec., zstd -6 --ultra --single-thread
     92,859,347 bytes,    30.555 sec.-      3.870 sec., 7z -mx3 -tbzip2 -mmt1
     92,810,578 bytes,    31.959 sec.-      4.070 sec., 7z -mx5 -tbzip2 -mmt1
     91,930,200 bytes,    79.564 sec.-      3.993 sec., 7z -mx7 -tbzip2 -mmt1
     89,292,399 bytes,   272.160 sec.-      3.558 sec., 7z -mx9 -tbzip2 -mmt1
     88,245,920 bytes,     4.538 sec. -     0.827 sec., lzturbo -32 -b1024 -p1
     87,866,645 bytes,     3.886 sec.-      0.433 sec., zstd -7 --ultra --single-thread
     86,374,802 bytes,     4.649 sec.-      0.423 sec., zstd -8 --ultra --single-thread
     85,881,714 bytes,     5.487 sec.-      0.422 sec., zstd -9 --ultra --single-thread
     85,060,590 bytes,    34.308 sec. -    35.908 sec., ppmd e -m256 -o8
     84,861,877 bytes,     6.438 sec.-      0.421 sec., zstd -10 --ultra --single-thread
     84,634,980 bytes,     7.789 sec.-      0.421 sec., zstd -11 --ultra --single-thread
     84,418,783 bytes,    17.501 sec.-      0.413 sec., zstd -13 --ultra --single-thread
     84,384,933 bytes,    10.286 sec.-      0.418 sec., zstd -12 --ultra --single-thread
     84,246,538 bytes,     5.558 sec.-      1.210 sec., rar -m1 -mt1
     84,171,820 bytes,    19.503 sec.-      0.414 sec., zstd -14 --ultra --single-thread
     84,043,256 bytes,    26.311 sec.-      0.415 sec., zstd -15 --ultra --single-thread
     80,871,756 bytes,    41.564 sec.-      0.427 sec., zstd -16 --ultra --single-thread
     79,062,917 bytes,     8.893 sec.-      0.952 sec., brotli_gc82 -q 5 --large_window=30
     78,796,316 bytes,    20.140 sec. -     6.911 sec., bwtturbo -59 -t0
     78,411,587 bytes,    11.515 sec.-      0.944 sec., brotli_gc82 -q 6 --large_window=30
     78,278,819 bytes,    49.844 sec.-      0.430 sec., zstd -17 --ultra --single-thread
     78,157,770 bytes,   130.282 sec. -   150.564 sec., nz -cc -t1
     77,638,400 bytes,    18.573 sec.-      0.936 sec., brotli_gc82 -q 7 --large_window=30
     77,383,789 bytes,    33.179 sec.-      0.933 sec., brotli_gc82 -q 8 --large_window=30
     76,807,130 bytes,    28.521 sec. -    34.126 sec., bcm -9
     76,631,786 bytes,    73.061 sec.-      0.932 sec., brotli_gc82 -q 9 --large_window=30
     75,951,871 bytes,   446.226 sec. -   348.386 sec., bbb cm1000
     75,824,086 bytes,    25.195 sec. -    20.394 sec., bsc -m0 -b1024 -e2 -T
     75,582,037 bytes,    69.137 sec.-      0.453 sec., zstd -18 --ultra --single-thread
     75,073,993 bytes,    86.311 sec.-      0.454 sec., zstd -19 --ultra --single-thread
     74,969,276 bytes,     9.114 sec.-      2.189 sec., 7z -mx1 -t7z -mmt1
     73,290,454 bytes,    13.004 sec.-      2.070 sec., 7z -mx3 -t7z -mmt1
     72,543,276 bytes,   247.662 sec. -     4.239 sec., glza -t1
     72,486,000 bytes,    93.220 sec.-      0.481 sec., zstd -20 --ultra --single-thread
     70,969,356 bytes,   103.202 sec.-      0.490 sec., zstd -21 --ultra --single-thread
     70,048,687 bytes,     8.352 sec.-      1.167 sec., rar -m2 -mt1
     69,480,082 bytes,   117.975 sec.-      0.501 sec., zstd -22 --ultra --single-thread
     67,674,309 bytes,     9.390 sec.-      1.136 sec., rar -m3 -mt1
     67,362,088 bytes,    13.057 sec.-      1.127 sec., rar -m4 -mt1
     67,243,590 bytes,    14.647 sec.-      1.122 sec., rar -m5 -mt1
     66,556,038 bytes,   450.634 sec.-      1.047 sec., brotli_gc82 -q 10 --large_window=30
     64,798,846 bytes,    80.630 sec.-      1.964 sec., 7z -mx5 -t7z -mmt1
     62,359,340 bytes,   100.765 sec.-      1.928 sec., 7z -mx7 -t7z -mmt1
     62,098,925 bytes,   690.067 sec.-      0.949 sec., brotli_gc82 -q 11 --large_window=30
     61,184,161 bytes,   107.507 sec.-      1.912 sec., 7z -mx9 -t7z -mmt1
     57,126,873 bytes,    56.055 sec. -    54.751 sec., zcmx64 -m8 -t1   
     53,090,885 bytes,    77.583 sec. -   xxx.xxx sec., mcm_generic -x -m11 (decomp take to long)
     52,847,099 bytes,   432.018 sec. -     2.855 sec., rz -d 1023M
     52,546,609 bytes,   560.550 sec. -   xxx.xxx sec., zpaq64 -m511 -t1
     47,100,239 bytes,  1707.788 sec. -  xxxx.xxx sec., fp8_v3 -8

    Input:
    399,998,976 bytes - block40.dat (Test 4 - roughly 30-70 mixture of Test 1 and Test 3 data)

    Output:
    Code:
    184,854,213 bytes,     0.650 sec.-      0.263 sec., lz4 -1
    155,565,151 bytes,     3.138 sec.-      0.258 sec., lz4 -3
    152,356,335 bytes,     4.384 sec.-      0.255 sec., lz4 -5
    151,191,133 bytes,     5.903 sec.-      0.254 sec., lz4 -7
    150,812,547 bytes,     8.557 sec.-      0.254 sec., lz4 -9
    149,491,434 bytes,     1.217 sec.-      1.137 sec., brotli_gc82 -q 0 --large_window=30
    139,468,782 bytes,     4.955 sec.-      2.129 sec., gzip -1
    138,482,759 bytes,     1.462 sec.-      1.078 sec., brotli_gc82 -q 1 --large_window=30
    136,580,784 bytes,     1.065 sec.-      0.464 sec., zstd -1 --ultra --single-thread
    132,017,337 bytes,     6.185 sec.-      2.031 sec., gzip -3
    129,446,737 bytes,     1.359 sec.-      0.484 sec., zstd -2 --ultra --single-thread
    124,383,697 bytes,     8.262 sec.-      2.005 sec., gzip -5
    124,138,139 bytes,     1.766 sec.-      0.492 sec., zstd -3 --ultra --single-thread
    122,906,961 bytes,     1.884 sec.-      0.518 sec., zstd -4 --ultra --single-thread
    122,531,497 bytes,    14.341 sec.-      1.975 sec., gzip -7
    121,946,034 bytes,    60.951 sec.-      1.964 sec., gzip -9
    121,728,460 bytes,     3.038 sec.-      0.924 sec., brotli_gc82 -q 2 --large_window=30
    120,421,936 bytes,     2.859 sec.-      0.522 sec., zstd -5 --ultra --single-thread
    120,133,711 bytes,     4.091 sec.-      1.097 sec., brotli_gc82 -q 3 --large_window=30
    119,142,375 bytes,     3.496 sec.-      0.520 sec., zstd -6 --ultra --single-thread
    117,361,115 bytes,     5.757 sec.-      1.034 sec., brotli_gc82 -q 4 --large_window=30
    114,276,021 bytes,     5.186 sec.-      0.489 sec., zstd -7 --ultra --single-thread
    113,460,257 bytes,    27.352 sec.-      3.383 sec., 7z -mx1 -tbzip2 -mmt1
    113,254,361 bytes,     6.420 sec.-      0.478 sec., zstd -8 --ultra --single-thread
    112,657,116 bytes,     7.839 sec.-      0.477 sec., zstd -9 --ultra --single-thread
    112,566,886 bytes,    29.905 sec.-      4.169 sec., 7z -mx3 -tbzip2 -mmt1
    111,819,082 bytes,    31.328 sec.-      4.435 sec., 7z -mx5 -tbzip2 -mmt1
    111,727,780 bytes,    83.494 sec.-      4.366 sec., 7z -mx7 -tbzip2 -mmt1
    111,076,866 bytes,     9.564 sec.-      0.479 sec., zstd -10 --ultra --single-thread
    110,377,975 bytes,    12.005 sec.-      0.481 sec., zstd -11 --ultra --single-thread
    110,174,286 bytes,   291.329 sec.-      3.867 sec., 7z -mx9 -tbzip2 -mmt1
    110,096,027 bytes,     5.556 sec. -     0.933 sec., lzturbo -32 -b1024 -p1
    109,989,855 bytes,    16.457 sec.-      0.479 sec., zstd -12 --ultra --single-thread
    109,559,003 bytes,    22.140 sec.-      0.473 sec., zstd -13 --ultra --single-thread
    108,417,157 bytes,    26.884 sec.-      0.479 sec., zstd -14 --ultra --single-thread
    107,712,708 bytes,    35.516 sec.-      0.483 sec., zstd -15 --ultra --single-thread
    105,665,543 bytes,    10.864 sec.-      1.093 sec., brotli_gc82 -q 5 --large_window=30
    104,533,189 bytes,   153.726 sec. -   125.871 sec., nz -cc -t1
    104,259,000 bytes,    14.040 sec.-      1.074 sec., brotli_gc82 -q 6 --large_window=30
    103,815,542 bytes,    45.487 sec.-      0.486 sec., zstd -16 --ultra --single-thread
    102,667,031 bytes,    22.259 sec.-      1.059 sec., brotli_gc82 -q 7 --large_window=30
    101,571,387 bytes,    11.471 sec.-      2.680 sec., 7z -mx1 -t7z -mmt1
    101,139,011 bytes,    38.302 sec.-      1.077 sec., brotli_gc82 -q 8 --large_window=30
    100,944,178 bytes,    60.172 sec.-      0.488 sec., zstd -17 --ultra --single-thread
     99,411,048 bytes,    82.571 sec.-      1.058 sec., brotli_gc82 -q 9 --large_window=30
     98,638,685 bytes,    78.266 sec.-      0.511 sec., zstd -18 --ultra --single-thread
     98,385,901 bytes,    18.605 sec.-      2.450 sec., 7z -mx3 -t7z -mmt1
     97,641,046 bytes,    93.578 sec.-      0.521 sec., zstd -19 --ultra --single-thread
     93,821,193 bytes,   109.980 sec.-      0.560 sec., zstd -20 --ultra --single-thread
     93,575,920 bytes,    40.051 sec. -    41.667 sec., ppmd -m256 -o8
     91,460,782 bytes,   124.530 sec.-      0.568 sec., zstd -21 --ultra --single-thread
     89,314,971 bytes,   140.094 sec.-      0.581 sec., zstd -22 --ultra --single-thread
     85,534,704 bytes,   104.134 sec.-      2.217 sec., 7z -mx5 -t7z -mmt1
     84,246,538 bytes,     5.558 sec.-      1.210 sec., rar -m1 -mt1
     83,980,924 bytes,   477.131 sec.-      1.223 sec., brotli_gc82 -q 10 --large_window=30
     82,928,976 bytes,    23.939 sec. -     7.xxx sec., bwtturbo -59 -t0 (decomp crash)
     82,861,609 bytes,   125.527 sec.-      2.203 sec., 7z -mx7 -t7z -mmt1
     80,927,550 bytes,   137.540 sec.-      2.209 sec., 7z -mx9 -t7z -mmt1
     80,635,084 bytes,    32.600 sec. -    37.356 sec., bcm -9
     80,511,030 bytes,   222.754 sec. -     6.201 sec., glza -t1
     80,249,048 bytes,    25.195 sec. -    20.394 sec., bsc -m0 -b1024 -e2 -T
     80,118,058 bytes,   372.074 sec. -   355.784 sec., bbb cm1000
     79,968,657 bytes,   734.674 sec.-      1.147 sec., brotli_gc82 -q 11 --large_window=30
     70,971,803 bytes,    68.454 sec. -    67.543 sec., zcmx64 -m8 -t1
     70,048,687 bytes,     8.352 sec.-      1.167 sec., rar -m2 -mt1
     69,992,690 bytes,   510.133 sec. -     3.691 sec., rz -d 1023M
     67,674,309 bytes,     9.390 sec.-      1.136 sec., rar -m3 -mt1
     67,362,088 bytes,    13.057 sec.-      1.127 sec., rar -m4 -mt1
     67,243,590 bytes,    14.647 sec.-      1.122 sec., rar -m5 -mt1
     65,152,519 bytes,   582.133 sec. -   xxx.xxx sec., zpaq64 -m511 -t1
     64,214,420 bytes,    85.848 sec. -   xxx.xxx sec., mcm_generic -x -m11 (decomp take to long)
     61,678,970 bytes,  1787.655 sec. -  xxxx.xxx sec., fp8_v3 -8
    Last edited by Sportman; 14th July 2020 at 22:58.

  2. Thanks:

    lz77 (9th July 2020)

  3. #2
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    326
    Thanks
    50
    Thanked 61 Times in 49 Posts
    81,046,263 bytes 1360.32 sec - 1175.06 sec bbbv1.9 cm1000
    Attached Files Attached Files

  4. #3
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    1,003
    Thanks
    97
    Thanked 399 Times in 278 Posts
    Quote Originally Posted by suryakandau@yahoo.co.id View Post
    81,046,263 bytes 1360.32 sec - 1175.06 sec bbbv1.9 cm1000
    Added.

  5. #4
    Member lz77's Avatar
    Join Date
    Jan 2016
    Location
    Russia
    Posts
    96
    Thanks
    27
    Thanked 13 Times in 9 Posts
    I'm interested in the time of compression + decompression on the I7 8700 CPU of the entire file. Let Sportman's computer 2 times faster than I7 8700. In this case on the Sportman's super computer time of compression + time of decompression must be smaller than 40 sec./2/2.5=8 sec. to meet the rapid compression test conditions.

    I do not see the result for lzturbo. Maybe it will be the best.

  6. #5
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    1,003
    Thanks
    97
    Thanked 399 Times in 278 Posts
    Quote Originally Posted by lz77 View Post
    Let Sportman's computer 2 times faster than I7 8700.
    i7-8700K:
    Max Turbo Frequency 4.70 GHz
    Intel Smart Cache 12 MB
    Memory Types DDR4-2666 (used 2400 MHz)

    i9-9900K:
    Max Turbo Frequency 5.00 GHz (used OC to 5.20 GHz)
    Intel Smart Cache 16 MB
    Memory Types DDR4-2666 (used 4266 MHz)

    My guess is 11.3% faster for CPU intensive tasks.

    Quote Originally Posted by lz77 View Post
    I do not see the result for lzturbo.
    bwtturbo -59 was tested.

  7. Thanks:

    lz77 (22nd June 2020)

  8. #6
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    326
    Thanks
    50
    Thanked 61 Times in 49 Posts
    Quote Originally Posted by Sportman View Post
    i7-8700K:
    Max Turbo Frequency 4.70 GHz
    Intel Smart Cache 12 MB
    Memory Types DDR4-2666 (used 2400 MHz)

    i9-9900K:
    Max Turbo Frequency 5.00 GHz (used OC to 5.20 GHz)
    Intel Smart Cache 16 MB
    Memory Types DDR4-2666 (used 4266 MHz)

    My guess is 11.3% faster for CPU intensive tasks.



    bwtturbo -59 was tested.


    I do not see the result for paq8sk23 -x15. Maybe it will be the best.

  9. #7
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    1,003
    Thanks
    97
    Thanked 399 Times in 278 Posts
    Quote Originally Posted by suryakandau@yahoo.co.id View Post
    I do not see the result for paq8sk23 -x15.
    I'm afraid you must make it first at least 36 times quicker...

    Compression + Decompression Time Limit, seconds:
    T1 High compression ratio, < 4000
    T2 High compression ratio, < 4000
    T3 High compression ratio, < 4000
    T4 High compression ratio, < 4000

  10. #8
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    326
    Thanks
    50
    Thanked 61 Times in 49 Posts
    Quote Originally Posted by Sportman View Post
    I'm afraid you must make it first at least 36 times quicker...

    Compression + Decompression Time Limit, seconds:
    T1 High compression ratio, < 4000
    T2 High compression ratio, < 4000
    T3 High compression ratio, < 4000
    T4 High compression ratio, < 4000
    Maybe you can test paq8sk19. It is the fastest version of paq8sk series

  11. #9
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    326
    Thanks
    50
    Thanked 61 Times in 49 Posts
    i have fixed a little in word model paq8pxd_89. @sportman could you test it using -x15 ? maybe it can compress better than the original
    Attached Files Attached Files

  12. #10
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,942
    Thanks
    291
    Thanked 1,286 Times in 728 Posts
    All paq8 versions are too slow for the competition - even fp8 (since TS40 is 40% of whole file and there's a 4000s time limit).

  13. #11
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    326
    Thanks
    50
    Thanked 61 Times in 49 Posts
    Quote Originally Posted by Shelwien View Post
    All paq8 versions are too slow for the competition - even fp8 (since TS40 is 40% of whole file and there's a 4000s time limit).
    So what algorithm suitable for this competition ?

  14. #12
    Member lz77's Avatar
    Join Date
    Jan 2016
    Location
    Russia
    Posts
    96
    Thanks
    27
    Thanked 13 Times in 9 Posts
    Quote Originally Posted by suryakandau@yahoo.co.id View Post
    So what algorithm suitable for this competition ?
    I think, compressors will have a size of 40MB and will contain a part of the file TS40.txt in a very compressed form. Or something like that.

  15. #13
    Member
    Join Date
    Jun 2020
    Location
    Belarus
    Posts
    6
    Thanks
    6
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Sportman View Post
    Input:
    400,000,000 bytes - TS40.txt (English texts from Project Gutenberg in UTF-8 )

    Output:
    How about UDA, maybe it also will compress within a time limit?

  16. #14
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    1,003
    Thanks
    97
    Thanked 399 Times in 278 Posts
    Quote Originally Posted by paleski View Post
    How about UDA
    Added.
    Last edited by Sportman; 24th June 2020 at 00:35.

  17. Thanks:

    paleski (24th June 2020)

  18. #15
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    1,003
    Thanks
    97
    Thanked 399 Times in 278 Posts
    Quote Originally Posted by Sportman View Post
    My guess is 11.3% faster for CPU intensive tasks.
    I tested PPMII variant J, May 20 2006:
    39.753 sec. - 40.490 sec., ppmd -m256 -o8 (default -r0)
    130.028 sec. - 132.782 sec., ppmd -m256 -o8 -r0 (https://globalcompetition.compression.ru/rules/)
    52.011 sec. - 53.113 sec. (divided by 1000MB/400MB=2.5)
    1.31x slower - 1.31x slower

    At 800 euro micro ATX desktop PC i7-8559U (4 core x 2.7GHz 4.5GHz turbo max, 2 x 8GB memory 2400MHz 14-14-14-35, 1TB NVMe 3500MB/s read, 3300MB/s write):
    51.427 sec. - 52.013 sec., ppmd -m256 -o8 (default -r0)
    1.29x slower - 1.28x slower

    Both systems use Windows 10 Pro, Power options at High performance, only needed services enabled and Meltdown and Spectre patches disabled.
    Last edited by Sportman; 25th June 2020 at 22:29.

  19. #16
    Member lz77's Avatar
    Join Date
    Jan 2016
    Location
    Russia
    Posts
    96
    Thanks
    27
    Thanked 13 Times in 9 Posts
    > 99,417,161 bytes, 12.430 sec. - 13.326 sec., ppmd -m256 -o4

    Is it (de)compressed in single thread??
    Last edited by lz77; 26th June 2020 at 09:45.

  20. #17
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    1,003
    Thanks
    97
    Thanked 399 Times in 278 Posts
    Quote Originally Posted by lz77 View Post
    >ppmd -m256 -o4
    Is it (de)compressed in single thread??
    Yes, both operations are single thread.

  21. Thanks:

    lz77 (26th June 2020)

  22. #18
    Member lz77's Avatar
    Join Date
    Jan 2016
    Location
    Russia
    Posts
    96
    Thanks
    27
    Thanked 13 Times in 9 Posts
    Oh, I see, a PPM algorithm can't be winner in Rapid compression:


    > 113,635,552 bytes, 10.122 sec. - 11.798 sec., ppmd -m256 -o3

    (10.122+11.79) *2.5*1.113 > 40 sec.


    Is zstd (LZ77+rANS?) to be winner in the list above?

    > 130,590,357 bytes, 10.530 sec. - 0.528 sec., zstd -8 --ultra --single-thread

  23. #19
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,942
    Thanks
    291
    Thanked 1,286 Times in 728 Posts
    > 113,635,552 bytes, 10.122 sec. - 11.798 sec., ppmd -m256 -o3
    > (10.122+11.79) *2.5*1.113 > 40 sec.

    Sure, but its possible (basically intended) to use preprocessing before compression,
    which would reduce 1Gb of text to around 600Mb (in case of WRT).

    > Is zstd (LZ77+rANS?) to be winner in the list above?

    zstd is LZ77+huffman mostly. It kinda uses tANS for matches, but FSE tANS isn't that much better than huffman.

    Also zstd encoding is slower than PPM at high levels, and isn't very good for text compression.

  24. Thanks:

    lz77 (27th June 2020)

  25. #20
    Member lz77's Avatar
    Join Date
    Jan 2016
    Location
    Russia
    Posts
    96
    Thanks
    27
    Thanked 13 Times in 9 Posts
    But ppmd in this list did not used text preprocessing??

  26. #21
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,942
    Thanks
    291
    Thanked 1,286 Times in 728 Posts
    > But ppmd in this list did not used text preprocessing??

    Its not, though Sportman could process TS40 with paq8pxd -s0 or something.

    I'm just saying that contest ranking of codecs is not that clear.

  27. Thanks:

    lz77 (28th June 2020)

  28. #22
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    1,003
    Thanks
    97
    Thanked 399 Times in 278 Posts
    Quote Originally Posted by Shelwien View Post
    though Sportman could process TS40 with paq8pxd -s0 or something.
    Added.

  29. Thanks:

    lz77 (28th June 2020)

  30. #23
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,942
    Thanks
    291
    Thanked 1,286 Times in 728 Posts
    paq8pxd -s0 is a preprocessor, I meant that it can be combined with ppmd and some other codecs that lack integrated preprocessing (eg. mcm has it).

  31. #24
    Member lz77's Avatar
    Join Date
    Jan 2016
    Location
    Russia
    Posts
    96
    Thanks
    27
    Thanked 13 Times in 9 Posts
    paq8pxd_v89_no_avx2 -s0 works 72 sec. Too slow preprocessing...
    I'm interested in the file size after maximum Rapid compression (with a condition of 40/2.5 sec. for compression + decompression).

  32. #25
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,942
    Thanks
    291
    Thanked 1,286 Times in 728 Posts
    > paq8pxd_v89_no_avx2 -s0 works 72 sec. Too slow preprocessing...

    Decoding is 24s though. Also there're others: xwrt, DRT, Shkarin's "liptify", Bulat's dict, cmix -s, mcm.
    Even WRT/LIPT is a simple algorithm, which has no reason to be slow (aside from inefficient implementation).

    > I'm interested in the file size after maximum Rapid compression (with a condition of 40/2.5 sec. for compression + decompression).

    Maybe try tweaking zstd? Like https://github.com/facebook/zstd/blo...s/paramgrill.c ?

  33. #26
    Member lz77's Avatar
    Join Date
    Jan 2016
    Location
    Russia
    Posts
    96
    Thanks
    27
    Thanked 13 Times in 9 Posts
    I think, lzturbo by DND after tuning has a good chance, but B. Ziganshin has a claim to this program... Has lzturbo the right to participate in the Competition?

  34. #27
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    326
    Thanks
    50
    Thanked 61 Times in 49 Posts
    Quote Originally Posted by suryakandau@yahoo.co.id View Post
    81,046,263 bytes 1360.32 sec - 1175.06 sec bbbv1.9 cm1000
    bbb v1.10
    400000000 -> 81046263 in 1071.57 sec
    faster than v1.9
    Attached Files Attached Files

  35. #28
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,187
    Thanks
    717
    Thanked 480 Times in 370 Posts
    paq8sk30 scores:

    70'587'620 - TS40.txt -x15 by Paq8sk30, time - 38'154,53s
    70'315'001 - TS40.txt -x15 -e1,english.dic by Paq8sk30, time - 45'253,18s
    72'656'037 - TS40.txt -x15 -w -e1,english.dic by Paq8sk30, time - 89'471,51s - worse score, bad time, I don't made decompression test...
    Last edited by Darek; 1st July 2020 at 11:51.

  36. Thanks:

    suryakandau@yahoo.co.id (30th June 2020)

  37. #29
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    326
    Thanks
    50
    Thanked 61 Times in 49 Posts
    Quote Originally Posted by Darek View Post
    paq8sk30 scores:

    70'587'620 - TS40.txt -x15 by Paq8sk30, time - 38'154,53s
    70'315'001 - TS40.txt -x15 -e1,english.dic by Paq8sk30, time - 45'253,18s
    72'656'037 - TS40.txt -x15 -w -e1,english.dic by Paq8sk30, time - 89'471,51s - worse score, bad time, I don't made decompression test...
    so the best score is using -x15 -e1,english.dic. @sportman could you add it to GDCC public test set file ?

  38. #30
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    326
    Thanks
    50
    Thanked 61 Times in 49 Posts
    Quote Originally Posted by Darek View Post
    paq8sk30 scores:

    70'587'620 - TS40.txt -x15 by Paq8sk30, time - 38'154,53s
    70'315'001 - TS40.txt -x15 -e1,english.dic by Paq8sk30, time - 45'253,18s
    72'656'037 - TS40.txt -x15 -w -e1,english.dic by Paq8sk30, time - 89'471,51s - worse score, bad time, I don't made decompression test...
    paq8pxd_v89 when using -w option cause error message
    Transform fails at 333440671, skipping...
    so it detect ts40.txt as default , not bigtext wrt then that cause the compression ratio is worse

Page 1 of 2 12 LastLast

Similar Threads

  1. IPC test set
    By FatBit in forum Data Compression
    Replies: 4
    Last Post: 23rd December 2013, 10:57
  2. Test set: bookstar
    By m^2 in forum Data Compression
    Replies: 5
    Last Post: 11th February 2009, 16:49
  3. Test set: installer
    By m^2 in forum Data Compression
    Replies: 10
    Last Post: 11th February 2009, 13:47
  4. Test set: music
    By m^2 in forum Data Compression
    Replies: 0
    Last Post: 9th December 2008, 23:25
  5. Test set: backup
    By m^2 in forum Data Compression
    Replies: 1
    Last Post: 23rd October 2008, 22:16

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •