Compressing millions of html

[2pact]

Hi there,
I need suggestions to choose the right software & algorithm for my case.
I have millions of .html files of archiving websites, each html file mostly contains text and few small images.
I need a compression solution that can provide fast compression and decompression rates to such files.

I have tested .7z, and it was a nightmare. any suggestion?

[SolidComp]

Take some of your files and generate a dictionary with Zstd then compress.

Brotli is also good with older HTML files.

[2pact]

Thanks, I think Zstd will do, it's dictionary creation is helpful.

[olavrb]

Also worth checking if those images can be losslessly compressed. Try "Pinga" for instance, will do JPG and PNG.

[Sportman]

Input:
861,938,525 bytes - 20,000 HTML pages + headers

Output:

107,282,956 bytes, 26.694 sec.,  1.015 + xx.xxx sec., xwrt -l0 -b255 +d & lzturbo -32 -b1024
115,966,329 bytes, 28.265 sec.,  1.976 + xx.xxx sec., xwrt -l0 -b255 +d & tor -6
116,778,388 bytes, 21.142 sec.,  2.127 sec., tor -7 -b1024
117,479,089 bytes, 25.799 sec.,  2.454 sec., tor-small -9 -b1024
117,691,077 bytes, 21.339 sec.,  2.051 sec., tor -7
118,861,524 bytes, 26.275 sec.,  2.445 sec., tor-small -9
119,356,953 bytes,  7.478 sec.,  1.312 sec., lzturbo -32 -b1024  
120,900,667 bytes, 22.213 sec.,  2.521 sec., tor-small -8 -b1024
121,138,361 bytes, 27.850 sec.,  2.386 + xx.xxx sec., xwrt -l0 -b255 +d & tor-small -6 -b1024
124,452,325 bytes, 22.891 sec.,  2.476 sec., tor-small -8
126,158,142 bytes, 15.736 sec.,  2.620 sec., tor-small -7 -b1024
127,266,149 bytes, 22.487 sec.,  3.690 + xx.xxx sec., xwrt -l0 -b255 +d & nz -cF -m1024m
127,635,533 bytes, 10.436 sec.,  2.273 sec., tor -6 -b1024
129,992,276 bytes, 15.764 sec.,  2.720 sec., nz -cdP -m1024m
131,064,250 bytes, 16.531 sec.,  2.526 sec., tor-small -7
131,918,731 bytes, 15.605 sec.,  2.606 sec., nz -cdP
132,919,600 bytes, 22.599 sec.,  1.568 sec., brotli -q 6 -w 24
133,267,492 bytes, 11.009 sec.,  2.214 sec., tor -6
133,751,721 bytes,  7.483 sec.,  1.322 sec., lzturbo -32
134,896,907 bytes, 10.145 sec.,  2.707 sec., tor-small -6 -b1024
136,558,579 bytes, 30.169 sec.,  0.861 sec., zstd -13
137,484,365 bytes, 28.750 sec., xx.xxx sec., xwrt -l2 -b255 +d
137,520,471 bytes, 28.605 sec., xx.xxx sec., xwrt -l2 -b255
137,523,947 bytes, 17.004 sec.,  1.617 sec., brotli -q 5 -w 24
137,655,975 bytes, 16.283 sec.,  0.813 sec., zstd -12
138,291,201 bytes, 26.825 sec.,  3.710 sec., 7z -mx4 -t7z
139,746,607 bytes, 26.719 sec.,  2.027 + xx.xxx sec., xwrt -l0 -b255 +d & rar -m1 -ma5
140,307,711 bytes, 10.943 sec.,  2.403 sec., tor-small -6
140,537,389 bytes, 14.895 sec., 20.804 sec., arc -m2
140,944,857 bytes, 14.398 sec.,  0.835 sec., zstd -11
141,165,390 bytes,  9.065 sec.,  1.507 sec., brotli -q 4 -w 24
141,226,215 bytes, 25.752 sec.,  1.960 + xx.xxx sec., xwrt -l0 -b255 +d & 7z -mx1 -tzip
141,452,371 bytes, 11.362 sec.,  0.838 sec., zstd -10
142,207,481 bytes,  9.068 sec.,  0.876 sec., zstd -9
143,480,522 bytes,  7.301 sec.,  0.835 sec., zstd -8
143,613,139 bytes, 19.290 sec.,  3.925 sec., 7z -mx3 -t7z
144,142,239 bytes,  6.922 sec.,  0.842 sec., zstd -7
147,815,180 bytes,  4.905 sec.,  0.863 sec., zstd -6
151,980,743 bytes, 18.480 sec.,  4.287 sec., 7z -mx2 -t7z
153,003,725 bytes,  2.784 sec.,  3.105 sec., nz -cF -m1024m
153,295,198 bytes,  2.676 sec.,  3.060 sec., nz -cF
156,119,324 bytes,  4.009 sec.,  0.896 sec., zstd -5
159,683,217 bytes,  5.575 sec.,  1.585 sec., brotli -q 3 -w 24
160,316,452 bytes, 17.487 sec.,  4.590 sec., 7z -mx1 -t7z
162,791,690 bytes,  4.888 sec.,  1.634 sec., brotli -q 2 -w 24
163,971,199 bytes,  2.423 sec.,  0.902 sec., zstd -4
164,415,042 bytes,  2.312 sec.,  0.897 sec., zstd -3
169,893,453 bytes,  9.400 sec.,  2.588 sec., rar -m1 -ma5
174,878,733 bytes,  1.911 sec.,  0.890 sec., zstd -2
179,431,420 bytes,  9.460 sec.,  2.613 sec., 7z -mx1 -tzip
181,138,030 bytes,  1.478 sec.,  0.869 sec., zstd -1
204,712,777 bytes,  2.869 sec.,  1.872 sec., brotli -q 1 -w 24
482,066,186 bytes, 19.271 sec., xx.xxx sec., xwrt -l0 -b255 +d

tor 0.5
lzturbo 1.2
tor-small 0.4a
brotli 1.0.4
zstd 1.3.5
nanozip 0.09
7-zip 18.05
freearc 0.67 (March 15 2014)
rar 5.60
xwrt 3.4

All tests single core except zstd decompression.

[2pact]

Your benchmark results are truly helpful. Thanks!!!!!!!!!
Also, for zstd did you create a dictionary?

[WinnieW]

What is tor-small? Never heard of it.

[Sportman]

for zstd did you create a dictionary?

No, I used default settings.

[Sportman]

What is tor-small? Never heard of it.

It was located in Bulat's standalone Tornado archive in the time freearc.org was up.

Version 0.4:
https://web.archive.org/web/20100922.../tornado04.zip

Version 0.4a (version I used):
https://web.archive.org/web/20140325...tornado04a.zip

Version 0.5 (only tor):
https://web.archive.org/web/20140325.../tornado05.zip

[Shelwien]

https://github.com/inikep/XWRT ?

[Sportman]

https://github.com/inikep/XWRT ?

Added.

[2pact]

I think mozjpeg is the best image compressor right now, atleast in my case.

[2pact]

What do you mean by xx.xxx sec for xwrt?

[Shelwien]

xwrt is actually a preprocessor, it can be used with any other codecs.
Sportman somehow only tested it with built-in zlib compression.
As to pictures (png/jpg), I'd suggest precomp; maybe even precomp/preflate.

[bwt]

Your benchmark results are truly helpful. Thanks!!!!!!!!!
Also, for zstd did you create a dictionary?

Maybe you want to try paq8px or paq8pxd or cmix...they are also in this forum...

[Sportman]

What do you mean by xx.xxx sec for xwrt?

I get xwrt decode error:
File corrupted (s.size()>WORD_MAX_SIZE)!
Not enough memory!

[Sportman]

Sportman somehow only tested it with built-in zlib compression.

Because I only include what compress till 30 seconds, otherwise it become to slow for very big HTML files or HTML big data sets.

Added XWRT & Lzturbo and XWRT & Tornado and XWRT & tor-small and XWRT & Nanozip and XWRT & RAR and XWRT & 7-zip ZIP.

[MegaByte]

I've had good success with mcm on log files (compared to 7zip ppmd); not sure how it would perform here.

[pothos2]

How about ZPAQ archiver http://mattmahoney.net/dc/zpaq.html which will dedupe and automatically choose a compression algorithm based on some short tests on the files.

[SolidComp]

XWRT is excellent and underexposed. It should've been the new web compressor (combined with a low overhead codec) instead of brotli. It beats brotli badly. It's weird that XWRT had just been sitting there for years and Google, Mozilla, Microsoft, and Apple ignored it.

Unless Zstd with a good HTML, CSS, JS dictionary beats XWRT.

[Jyrki Alakuijala]

XWRT is excellent and underexposed.

Is it streamable during decoding? What about encoding? How much ram does it use at decoding?

(Browser developers would not integrate non-streamable decoders or decoders that allocate uncontrolled or large amounts of ram.)

You are chaining it with other compressors -- what happens if you chain it with brotli or zstd?

Unless Zstd with a good HTML, CSS, JS dictionary beats XWRT.

Classic Brotli has user-defined dictionaries as well.

In Shared Brotli there is an even more powerful dictionary modeling as we have added the transform modeling and unique length-distance encoding to user-defined dictionaries as well (the mechanisms we use for static dictionary). These simple mods typically make user-defined LZ77 dictionaries about 10-20 % more powerful -- for example if you would get 10 % savings from a custom LZ77 dictionary, you can get 11-12 % savings from a custom Shared Brotli dictionary. (We also added context-modeling for reordering the dictionary for some anticipated further savings, but no quantitative results on it yet.)

[SolidComp]

Is it streamable during decoding? What about encoding? How much ram does it use at decoding?

(Browser developers would not integrate non-streamable decoders or decoders that allocate uncontrolled or large amounts of ram.)

You are chaining it with other compressors -- what happens if you chain it with brotli or zstd?



Classic Brotli has user-defined dictionaries as well.

In Shared Brotli there is an even more powerful dictionary modeling as we have added the transform modeling and unique length-distance encoding to user-defined dictionaries as well (the mechanisms we use for static dictionary). These simple mods typically make user-defined LZ77 dictionaries about 10-20 % more powerful -- for example if you would get 10 % savings from a custom LZ77 dictionary, you can get 11-12 % savings from a custom Shared Brotli dictionary. (We also added context-modeling for reordering the dictionary for some anticipated further savings, but no quantitative results on it yet.)

You're right that we should be able to chain it to brotli. On the brotli user-defined dictionaries, how do we create them? Where are the instructions? Zstd has the instructions for training, generating, and using a dictionary right there on their homepage, but with brotli all I saw were build instructions with premake and all that jazz. Where are the user docs?

[Sportman]

Input:
4,104,832,596 bytes, 50,000 HTML pages + headers
https://commoncrawl.s3.amazonaws.com...-00639.warc.gz

Output:

  638,628,491 bytes, 151.434 sec.,  9.531 sec., rar -m2 -ma5
  646,609,659 bytes,  33.043 sec.,  5.023 sec., lzturbo -32 -b1024
  649,879,350 bytes, 114.606 sec., 31.446 sec., arc -m3
  652,615,389 bytes, 176.503 sec.,  6.291 sec., brotli -q 7 -w 24
  675,786,114 bytes,  64.586 sec., 14.663 sec., arc -m2
  677,664,500 bytes, 154.458 sec.,  3.412 sec., zstd -14
  678,840,512 bytes,  85.041 sec.,  6.108 sec., brotli -q 6 -w 24
  678,960,551 bytes, 122.182 sec., 15.847 sec., 7z -mx4 -t7z
  679,967,079 bytes, 126.039 sec.,  3.445 sec., zstd -13
  683,375,896 bytes,  62.438 sec.,  3.476 sec., zstd -12
  684,497,537 bytes,  33.387 sec.,  4.478 sec., lzturbo -32
  692,992,930 bytes,  78.352 sec., 11.648 sec., nz -cdP -m1024m
  695,149,591 bytes,  63.323 sec.,  6.264 sec., brotli -q 5 -w 24
  697,952,704 bytes,  55.730 sec.,  3.471 sec., zstd -11
  699,764,448 bytes,  44.347 sec.,  3.464 sec., zstd -10
  701,111,732 bytes,  73.735 sec., 11.164 sec., nz -cdP
  701,702,322 bytes,  86.982 sec., 16.730 sec., 7z -mx3 -t7z
  702,414,691 bytes,  35.063 sec.,  3.492 sec., zstd -9
  707,020,525 bytes,  28.819 sec.,  3.461 sec., zstd -8
  709,451,933 bytes,  27.278 sec.,  3.486 sec., zstd -7
  722,729,124 bytes,  20.239 sec.,  3.601 sec., zstd -6
  734,191,921 bytes,  34.416 sec.,  6.434 sec., brotli -q 4 -w 24
  756,188,016 bytes,  12.405 sec., 13.413 sec., nz -cF -m1024m
  756,225,594 bytes,  87.151 sec., 18.418 sec., 7z -mx2 -t7z
  757,961,627 bytes,  12.209 sec., 13.228 sec., nz -cF
  759,237,267 bytes,  16.883 sec.,  3.685 sec., zstd -5
  763,168,591 bytes,  23.112 sec.,  6.472 sec., brotli -q 3 -w 24
  776,060,002 bytes,  19.364 sec.,  6.648 sec., brotli -q 2 -w 24
  788,313,067 bytes,   9.536 sec.,  3.754 sec., zstd -4
  790,161,648 bytes,   9.266 sec.,  3.685 sec., zstd -3
  818,253,204 bytes,  42.320 sec., 11.156 sec., rar -m1 -ma5
  843,409,165 bytes,  86.456 sec., 21.271 sec., 7z -mx1 -t7z
  851,100,961 bytes,   7.765 sec.,  3.753 sec., zstd -2
  878,916,253 bytes,   5.995 sec.,  3.652 sec., zstd -1
  930,837,437 bytes, 122.964 sec., 14.869 sec., tor -7 -b1024
  952,798,194 bytes,  89.072 sec., 17.272 sec., tor-small -7 -b1024
  955,192,051 bytes,  45.026 sec., 12.088 sec., 7z -mx1 -tzip
  956,171,387 bytes,  60.056 sec., 15.273 sec., tor -6 -b1024
1,071,815,433 bytes,  12.787 sec.,  8.547 sec., brotli -q 1 -w 24

tor 0.5
lzturbo 1.2, beta, beta II
tor-small 0.4a
brotli 1.0.4
zstd 1.3.5
nanozip 0.09
7-zip 18.05
freearc 0.67 (March 15 2014)
rar 5.60

All tests single core except zstd decompression.

[Jyrki Alakuijala]

... but with brotli all I saw were build instructions with premake and all that jazz. Where are the user docs?

Would you consider filing an issue on the github repository about the unclarity?

[SolidComp]

Would you consider filing an issue on the github repository about the unclarity?

Done. https://github.com/google/brotli/issues/697

I know you've asked me to do this before -- sorry it took me so long...

[SolidComp]

Would you consider filing an issue on the github repository about the unclarity?

By the way, is the custom dictionary part of the payload of the compressed file, or it a standalone dictionary that the recipient needs to have in order to decompress the file?

[Sportman]

Input:
57,269,226,154 bytes, Alexa top 1,000,000 HTML pages + headers

Output:

 6,682,500,126 bytes, 2683.736 sec.,  134.560 sec., rar -m2 -ma5 -md1g
 7,203,605,682 bytes, 2398.672 sec., 1494.298 sec., bsc -m0
 7,406,872,586 bytes, 1208.081 sec., 2010.735 sec., bsc -m6
 7,439,216,266 bytes,  301.624 + 469.578 sec., 85.830 + 68.479 sec., srep -m0 + lzturbo -32 -b1024
 7,442,736,590 bytes,  596.676 sec.,   86.695 sec., lzturbo (beta II) -32 -b1024
 7,462,762,823 bytes,  594.145 sec.,   86.675 sec., lzturbo (beta) -32 -b1024
 7,529,586,920 bytes, 1091.866 sec., 1890.670 sec., bsc -m5
 7,681,446,412 bytes, 2645.614 sec., 2458.265 sec., arc -m3
 7,689,994,916 bytes, 2629.968 sec.,  254.009 sec., nz -cD -m1024m
 7,721,814,081 bytes,  486.150 sec.,   95.146 sec., lzturbo (v1.2) -32 -b1024
 7,744,316,303 bytes, 2073.887 sec.,  132.313 sec., rar -m2 -ma5
 7,751,700,878 bytes,  966.680 sec., 1738.634 sec., bsc -m4
 7,708,706,413 bytes, 2633.326 sec.,  257.811 sec., nz -cDP -m1024m
 7,819,349,725 bytes, 2484.101 sec.,  256.802 sec., nz -cDp -m1024m
 7,968,006,316 bytes, 2693.067 sec.,   93.510 sec., brotli -q 7 -w 24
 8,306,530,796 bytes, 2232.525 sec.,   53.801 sec., ztsd -14
 8,347,490,076 bytes, 1843.954 sec.,   54.127 sec., ztsd -13
 8,395,993,618 bytes,  987.160 sec., 1549.477 sec., bsc -m3
 8,402,717,390 bytes, 1336.677 sec.,   90.671 sec., brotli -q 6 -w 24
 8,402,867,094 bytes, 1628.416 sec.,  226.449 sec., 7z -mx4 -t7z
 8,408,749,340 bytes,  986.207 sec.,   55.216 sec., ztsd -12
 8,458,196,886 bytes,  895.097 sec., 1258.219 sec., arc -m2
 8,576,544,152 bytes,  883.037 sec.,   54.046 sec., ztsd -11
 8,603,328,832 bytes,  709.001 sec.,   54.126 sec., ztsd -10
 8,634,534,227 bytes, 1160.808 sec.,  237.742 sec., 7z -mx3 -t7z
 8,637,690,336 bytes,  970.064 sec.,   93.386 sec., brotli -q 5 -w 24
 8,643,759,419 bytes,  569.279 sec.,   54.046 sec., ztsd -9
 8,722,553,984 bytes,  457.780 sec.,   54.144 sec., ztsd -8
 8,945,785,642 bytes, 1459.253 sec.,  256.516 sec., nz -cdP -m1024m
 8,974,062,112 bytes, 1099.432 sec.,  251.994 sec., 7z -mx2 -t7z
 8,986,125,666 bytes,  311.708 sec.,   56.016 sec., ztsd -6
 9,020,251,494 bytes,  998.225 sec.,  257.318 sec., nz -cd -m1024m
 9,105,934,610 byte,  1310.830 sec.,  259.954 sec., nz -cdp -m1024m
 9,276,336,130 bytes,  495.980 sec.,   95.928 sec., brotli -q 4 -w 24
 9,367,172,809 bytes,  227.601 sec.,  242.428 sec., nz -cF -m1024m
 9,397,787,657 bytes,  203.182 sec.,   81.220 sec., lzturbo (beta II) -31 -b1024
 9,409,591,496 bytes,  941.024 sec.,  360.078 sec., zpaq -m1
 9,436,992,864 bytes,  251.616 sec.,   57.529 sec., ztsd -5
 9,474,960,075 bytes, 1051.552 sec.,  275.146 sec., 7z -mx1 -t7z
 9,493,448,784 bytes,  206.122 sec.,   81.006 sec., lzturbo (beta) -31 -b1024
 9,684,434,905 bytes,  358.149 sec.,   97.541 sec., brotli -q 3 -w 24
 9,861,326,399 bytes,  287.412 sec.,  146.402 sec., arc -m1
 9,912,692,170 bytes,  299.844 sec.,  101.052 sec., brotli -q 2 -w 24
 9,917,117,752 bytes,  151.228 sec.,   59.004 sec., ztsd -4
 9,940,159,062 bytes,  144.961 sec.,   57.875 sec., ztsd -3
10,230,258,393 bytes,  657.482 sec.,   52.604 sec., lzturbo (beta) -22 -b1024
10,404,975,395 bytes,  133.783 sec.,   85.550 sec., lzturbo (beta II) -30 -b1024
10,421,145,769 bytes,  589.954 sec.,  157.898 sec., rar -m1 -ma5 -md1g
10,425,235,935 bytes,  588.781 sec.,  158.155 sec., rar -m1 -ma5
10,464,649,227 bytes,  624.864 sec.,   57.328 sec., ztsd -5 --format=gzip
10,549,597,390 bytes,  120.453 sec.,   58.298 sec., ztsd -2
10,605,605,829 bytes, 1141.219 sec.,  205.593 sec., tor-small -7 -b1024
10,688,066,823 bytes,  141.622 sec.,   84.939 sec., lzturbo (beta) -30 -b1024
10,727,646,134 bytes,  580.659 sec.,  159.395 sec., 7z -mx1 -tzip
10,913,491,962 bytes,   94.356 sec.,   57.048 sec., ztsd -1
11,354,422,547 bytes,  196.824 sec.,   55.218 sec., lzturbo (beta) -21 -b1024
11,629,817,134 bytes,  164.313 sec.,  174.498 sec., nz -cf -m1024m
12,189,230,146 bytes,  336.405 sec.,   57.918 sec., ztsd -1 --format=gzip
12,378,919,264 bytes,  174.560 sec.,  115.417 sec., brotli -q 1 -w 24
12,791,392,694 bytes,  148.285 sec.,  121.207 sec., brotli -q 0 -w 24
12,941,717,182 bytes,  623.304 sec.,   52.116 sec., lzturbo (beta) -12 -b1024
12,966,122,553 bytes,  123.862 sec.,   59.483 sec., lzturbo (beta) -20 -b1024
13,774,047,393 bytes,  185.359 sec.,   54.117 sec., lzturbo (beta) -11 -b1024
14,558,206,392 bytes,  126.632 sec.,   55.053 sec., lzturbo (beta) -10 -b1024
46,936,255,308 bytes,  301.624 sec.,   68.479 sec., srep -m0

bsc 3.1.0
lzturbo 1.2 + beta
tor-small 0.4a
brotli 1.0.4
zstd 1.3.5
nanozip 0.09
7-zip 18.05
freearc 0.67 (March 15, 2014)
rar 5.60
zpaq 7.15
srep 3.93a beta (October 11, 2014)

All tests single core except zstd decompression.

[kaitz]

How srep performs on this set? +some compressor

[Sportman]

How srep performs on this set? +some compressor

Added.

[Shelwien]

rar results seem pretty good... I want to suggest adding some options:

327,445,811 4.rar // rar560 a -ma5 -md1g 4.rar !arc
275,572,136 3.rar // rar560 a -ma5 -md1g -oi1 3.rar !arc
275,499,739 2.rar // rar560 a -ma5 -md1g -oi1 -ts- 2.rar !arc
275,452,675 1.rar // rar560 a -ma5 -md1g -oi1 -ts- -qo- 1.rar !arc