RAZOR - strong LZ-based archiver

[SvenBent]

I deleted the test files before I realized that I could "set temp=D:\".

I would rather use "-w disk:\someplace\" where RZ put the temp files, or even better - using only RAM, which in my case becomes impossible since this test computer only have 4GB.



C: is to small.

You can remove you temp folder permantly to another drive if space is an issue.

[a902cd23]

Hi,

how do you set extraction path (destination)?

I tried rz x archive.rz C:\users\user\output
without success. I tried "e" instead of "x"

rz.exe x -o DeStInAt.IoN\ ArChIvE.rz files*

[ScottX]

Hi,

how do you set extraction path (destination)?

I tried rz x archive.rz C:\users\user\output
without success. I tried "e" instead of "x"

rz.exe x -o <output directory> ...

[bigcrown]

Absolutely!

This is certainly the next-generation archiving platform.

For almost two decades (easily 15 years), the best we had was 7-Zip.

We all owe Igor Pavlov a huge thanks for making all of his work open source!

With all our applause, he seems to be handing the crown over to Razor now.

No way. NanoZip 0.09 alpha exceed RAZOR. With 10 times faster packing speed (multithread) it makes same packing ratio. With ultimate compression (-cc) it exceed RAZOR in packing ratio.
Tested on i7-4790K with 16Gb DDR3-1600 RAM.
Packing speed default 10000Kb/s NanoZip 900-1100Kb/s RAZOR
NanoZip ultimate (-cc) - 1200Kb/s

Your comments?

[Raider56]

No way. NanoZip 0.09 alpha exceed RAZOR. With 10 times faster packing speed (multithread) it makes same packing ratio. With ultimate compression (-cc) it exceed RAZOR in packing ratio.
Tested on i7-4790K with 16Gb DDR3-1600 RAM.
Packing speed default 10000Kb/s NanoZip 900-1100Kb/s RAZOR
NanoZip ultimate (-cc) - 1200Kb/s

Your comments?

Maybe consider decompression speed too ?
NanoZip's so called ultimate in multi-threading = almost 100x slower than Razor (1 thread) on average PCs
NanoZip's -cO (beats Razor in compression in some cases) in mt = 10x slower than Razor (1 thread) on average PCs or even slower than that

All the rest settings of nanozip cannot beat Razor. -cc beats Razor while -cO beats Razor only in some cases, mostly Razor wins. As for packing speed, I think Christian said that he will introduce multi-threading.

[ScottX]

No way. NanoZip 0.09 alpha exceed RAZOR. With 10 times faster packing speed (multithread) it makes same packing ratio. With ultimate compression (-cc) it exceed RAZOR in packing ratio.
Tested on i7-4790K with 16Gb DDR3-1600 RAM.
Packing speed default 10000Kb/s NanoZip 900-1100Kb/s RAZOR
NanoZip ultimate (-cc) - 1200Kb/s

Your comments?

It depends, what is important for you - compression or decompression speed. NanoZip -cc, -cO, -co methods have slower decompression speed than Razor. Razor is unique in great compression ratio and extraction speed.

Source: 980MB of data (game):

-----------------------------------------------------------------
| Compressor | Method   | Compressed | Comp. speed | Dec. speed |
|------------+----------+------------+-------------+------------|
| Razor      |          |     540 MB |      1 MB/s |    68 MB/s |
| NanoZip    |  -cO -p6 |     543 MB |      6 MB/s |     9 MB/s |
| NanoZip    |  -co -p6 |     549 MB |     13 MB/s |    21 MB/s |
| NanoZip    | -cDP -p6 |     587 MB |     67 MB/s |   587 MB/s |
-----------------------------------------------------------------

[Sportman]

Input:
3,203,068,928 bytes - Bitcoin blockchain all chainstates as tar.

Output:
2,013,774,071 bytes, 3677.531 sec., 46.907 sec., RAZOR - -d 1023M
2,108,324,432 bytes, 8408.080 sec., 8552.209 sec., Zpaq - 510
2,177,335,991 bytes, 1638.819 sec., 1399.067 sec., ZCM - 8
2,217,516,962 bytes, 1344.957 sec., 108.548 sec., FreeArc - 9
2,246,581,553 bytes, 3015.764 sec., 45.636 sec., RAZOR - default
2,247,740,569 bytes, 2335.628 sec., 795.172 sec., NanoZip - O
2,256,918,857 bytes, 2643.591 sec., 2653.724 sec., NanoZip - c
2,296,309,729 bytes, 959.352 sec., 4.856 sec., Zstd - 22
2,308,864,303 bytes, 1302.764 sec., 49.418 sec., 7-Zip - 9
2,337,987,050 bytes, 1466.707 sec., 1477.968 sec., MCM - x
2,413,429,564 bytes, 6794.124 sec., 17.959 sec., Brotli - 11
2,430,873,070 bytes, 137.313 sec., 107.943 sec., Lrzip - 9
2,446,470,888 bytes, 806.603 sec., 20.986 sec., WinRAR - 5
2,549,973,980 bytes, 622.827 sec., 424.239 sec., Bsc - 6
2,733,804,570 bytes, 4093.027 sec., 76.965 sec., Bzip2 - 9
2,740,972,403 bytes, 906.456 sec., 17.760 sec., Zip - 9

[jethro]

Input:
3,203,068,928 bytes - Bitcoin blockchain all chainstates as tar.

Output:
2,013,774,071 bytes, 6580.625 sec. - 51.756 sec., RAZOR - -d 1023M
2,217,516,962 bytes, 1340.715 sec. - 101.610 sec., FreeArc - 9
2,232,978,859 bytes, 1693.085 sec. - 1338.452 sec., ZCM - 7
2,256,918,857 bytes, 2737.249 sec. - 2871.098 sec., NanoZip - c
2,308,864,303 bytes, 1448.216 sec. - 50.415 sec., 7-Zip - 9
2,446,470,888 bytes, 880.098 sec. - 21.791 sec., WinRAR - 5
2,549,973,980 bytes, 114.486 sec. - 63.250 sec., Bsc - 6

Hello Sportsman,
Can you add mcm (skbuild) to this comparison?

[Sportman]

Can you add mcm (skbuild) to this comparison?

Done.

[jethro]

Done.

Thanks. Below my expectations. The speed is good but not the compresssion ratio this time. For me MCM is the most practical CM compressor and it would seem it was not tuned for such atpical data.

[mhajicek]

@bigcrown: also don't forget that nanozip -cc mode still contains a compression bug that in some cases prevents correct decompression, so I really cant recommend it for real use, or at least u should test all archives u create carefully

@sportman: in your comparison, more correct zpaq option would be -510

[Zonder]

@bigcrown: also don't forget that nanozip -cc mode still contains a compression bug that in some cases prevents correct decompression, so I really cant recommend it for real use, or at least u should test all archives u create carefully

Same with nanozip -cO mode, some archives can't be decompressed, "Archive corrupted. Error decoding (code 106)".

[Sportman]

in your comparison, more correct zpaq option would be -510

Done.

[Sportman]

Delete

[frosticles]

Done.

Out of interest could you test RAZOR with its standard dictionary size as well? Just to compare what speed-ratio tradeoff that makes in this example.

[Sportman]

Out of interest could you test RAZOR with its standard dictionary size as well? Just to compare what speed-ratio tradeoff that makes in this example.

Done, more then double compression speed.

[Sportman]

Input:
618,960,704 bytes - Wikipedia database dump nowiki.sql external links

Output:
36,582,403 bytes, 106.719 sec., 97.622 sec., MCM - x
37,310,918 bytes, 439.080 sec., 437.802 sec., NanoZip - c
37,477,593 bytes, 68.432 sec., 61.205 sec., ZCM - 8
37,913,072 bytes, 978.707 sec., 941.568 sec., Zpaq - 510
39,951,850 bytes, 848.639 sec., 2.523 sec., RAZOR - -d 1023M
42,080,505 bytes, 691.572 sec., 2.362 sec., RAZOR - default
47,548,745 bytes, 479.545 sec., 2.067 sec., Lzturbo - 49
53,765,062 bytes, 305.370 sec., 3.729 sec., FreeArc - 9
57,332,607 bytes, 211.821 sec., 2.232 sec., 7-Zip - 9
58,482,357 bytes, 893.739 sec., 0.818 sec., Brotli - 11
59,078,436 bytes, 326.426 sec., 4.825 sec., Glza - default
60,168,146 bytes, 310.311 sec., 0.692 sec., Zstd - 22
61,526,960 bytes, 11.280 sec., 21.069 sec., Bsc - 6
65,745,355 bytes, 51.651 sec., 3.860 sec., Lrzip - 9
69,721,798 bytes, 105.216 sec., 1.370 sec., WinRAR - 5
79,124,039 bytes, 469.795 sec., 5.875 sec., Bzip2 - 9
88,160,584 bytes, 385.616 sec., 1.487 sec., Zip - 9

[JamesB]

I usually use -m0e2 for bsc; BWT rather than sort-transform when testing the power, and the adaptive QLFC entropy coder, possibly also upping the block size (-b100 or more), although -m6 is ideal if you're looking for speed. Eg with enwiki9:


I'm still very impressed with RAZOR though - it pushes the boundary of what is possible with LZ methods.

Default:     compressed 1000000000 into 195254122 in 12.139 seconds. (NB: 32-core system)
-m6:         compressed 1000000000 into 202545396 in 7.114 seconds. (NB: 32-core system)
-m0e2:       compressed 1000000000 into 193868110 in 13.875 seconds. (NB: 32-core system)
-m0e2pb1024: compressed 1000000000 into 163924696 in 214.334 seconds. (Block size means only 1 core in use)

Edit, although on a snippet of enwiki sql dump -m6 is indeed better than -m0.

[Sportman]

Edit, although on a snippet of enwiki sql dump -m6 is indeed better than -m0.

nowiki.sql:
62,407,472 bytes, 35.401 sec., 22.313 sec., Bsc - -m0e2
60,042,056 bytes, 48.935 sec., 25.975 sec., Bsc - -m0e2pb2014
59,716,790 bytes, 45.884 sec., 25.121 sec., Bsc - -m0e2b2014

[Sportman]

Input:
1,105,730,560 bytes - MongoDB 3.x latest binaries as tar

Output:
103,335,718 bytes, 1488.712 sec., 5.752 sec., RAZOR - -d 1023M
114,969,028 bytes, 2211.284 sec., 2177.389 sec., Zpaq - 510
126,988,406 bytes, 184.576 sec., 167.244 sec., ZCM - 8
127,579,809 bytes, 877.441 sec., 864.501 sec., NanoZip - c
130,735,482 bytes, 1226.455 sec., 6.153 sec., RAZOR - default
132,988,043 bytes, 225.735 sec., 231.151 sec., MCM - x
144,876,496 bytes, 422.534 sec., 9.001 sec., FreeArc - 9
153,700,868 bytes, 398.987 sec., 5.203 sec., 7-Zip - 9
153,724,450 bytes, 60.351 sec., 9.522 sec., Lrzip - 9
163,164,905 bytes, 2021.226 sec., 2.045 sec., Brotli - 11
176,482,303 bytes, 979.753 sec., 11.661 sec., Glza - default
178,904,236 bytes, 559.769 sec., 1.635 sec., Zstd - 22
195,832,216 bytes, 38.243 sec., 50.850 sec., Bsc - 6
202,761,888 bytes, 142.436 sec., 3.350 sec., WinRAR - 5
239,785,472 bytes, 935.558 sec., 12.998 sec., Bzip2 - 9
291,261,004 bytes, 484.457 sec., 4.059 sec., Zip - 9

[Cyan]

Do you have a link to download or generate tested file ?

[Sportman]

Do you have a link to download or generate tested file ?

Direct file download:
https://bitcoin.org/bin/bitcoin-core...in64-setup.exe (sync till block 516,392 then files in folder C:\Users\Username\AppData\Roaming\Bitcoin\chainsta te\)
https://dumps.wikimedia.org/nowiki/2...allinks.sql.gz
https://downloads.mongodb.org/win32/...3.6-latest.zip (April 24, 2018 download)

From folder:
https://bitcoin.org/bin/bitcoin-core-0.16.0/
https://dumps.wikimedia.org/nowiki/20180401/
https://www.mongodb.org/dl/win32/x86_64-2008plus-ssl

[Sportman]

Input:
623,108,642 bytes - IIS log file (one day traffic)

Output:
9,650,928 bytes, 70.633 sec., 66.694 sec., MCM - x
10,000,887 bytes, 36.907 sec., 32.967 sec., ZCM - 8
10,844,330 bytes, 905.773 sec., 905.606 sec., Zpaq - 510
11,125,396 bytes, 462.724 sec., 460.423 sec., NanoZip - c
14,742,538 bytes, 32.483 sec., 1.928 sec., Glza - default
15,348,149 bytes, 688.388 sec., 1.363 sec., RAZOR - -d 1023M
15,786,244 bytes, 631.425 sec., 1.324 sec., RAZOR - default
16,451,972 bytes, 16.667 sec., 11.403 sec., FreeArc - 9
17,928,821 bytes, 958.274 sec., 0.437 sec., Lzturbo - 49
20,441,700 bytes, 5.495 sec., 9.974 sec., Bsc - 6
22,566,228 bytes, 329.684 sec., 0.467 sec., Zstd - 22
24,191,897 bytes, 86.546 sec., 1.105 sec., 7-Zip - 9
24,686,883 bytes, 727.043 sec., 0.448 sec., Brotli - 11
25,920,694 bytes, 46.622 sec., 0.776 sec., WinRAR - 5
26,008,113 bytes, 145.501 sec., 3.474 sec., Lrzip - 9
27,004,788 bytes, 154.370 sec., 0.664 sec., CLZ - default
33,337,375 bytes, 455.380 sec., 6.452 sec., Bzip2 - 9
56,723,760 bytes, 233.731 sec., 1.131 sec., Zip - 9

Multi-threaded:
20,441,700 bytes, 1.367 sec., 2.149 sec., Bsc - 6
24,557,564 bytes, 33.087 sec., 0.722 sec., 7-Zip - 9
25,921,886 bytes, 9.232 sec., 0.636 sec., WinRAR - 5

[Christian]

Input:
1,105,730,560 bytes - MongoDB 3.x latest binaries as tar

Output:
103,335,718 bytes, 1488.712 sec., 5.752 sec., RAZOR - -d 1023M
114,969,028 bytes, 2211.284 sec., 2177.389 sec., Zpaq - 510
126,988,406 bytes, 184.576 sec., 167.244 sec., ZCM - 8
127,579,809 bytes, 877.441 sec., 864.501 sec., NanoZip - c
130,735,482 bytes, 1226.455 sec., 6.153 sec., RAZOR - default
132,988,043 bytes, 225.735 sec., 231.151 sec., MCM - x
144,876,496 bytes, 422.534 sec., 9.001 sec., FreeArc - 9
153,700,868 bytes, 398.987 sec., 5.203 sec., 7-Zip - 9
153,724,450 bytes, 60.351 sec., 9.522 sec., Lrzip - 9
163,164,905 bytes, 2021.226 sec., 2.045 sec., Brotli - 11
176,482,303 bytes, 979.753 sec., 11.661 sec., Glza - default
178,904,236 bytes, 559.769 sec., 1.635 sec., Zstd - 22
195,832,216 bytes, 38.243 sec., 50.850 sec., Bsc - 6
202,761,888 bytes, 142.436 sec., 3.350 sec., WinRAR - 5
239,785,472 bytes, 935.558 sec., 12.998 sec., Bzip2 - 9
291,261,004 bytes, 484.457 sec., 4.059 sec., Zip - 9

Hi Sportman,

thank you very much for your benchmarks! It's very motivating and interesting to see these results.

Almost always, if results are too good, something's not right. I have a feeling, that in most tests where razor dominates conveniently, it's because of the 1G dictionary. 7z, nanozip, zstd and the likes should be much closer with similar dictionaries - e.g. 7z can not apply a x86-transform on a tar. I assume, that nanozip handles tars just fine. Sami's archivers always had very advanced data detection.

I just did a 'quick' retest on MongoDB without tar:

name      dec-time   size        options
-----------------------------------------------
rz        8.7s       103636293   -d 1023m
nanozip   60s        108917410   -cO -m4g
7zip      8.4s       112280723   ultra 1G ws273

Still, a good result. But much closer than before.

...I'm still very impressed with RAZOR though - it pushes the boundary of what is possible with LZ methods...

Thank you. I think, we can push LZ/ROLZ even further without touching decompression speed. But all gains are bought by increasingly obscene amounts of encoding time.

Have a nice evening,

Christian

[Sportman]

I just did a 'quick' retest on MongoDB without tar:

name      dec-time   size        options
-----------------------------------------------
rz        8.7s       103636293   -d 1023m
nanozip   60s        108917410   -cO -m4g
7zip      8.4s       112280723   ultra 1G ws273

Still, a good result. But much closer than before.

Thanks, I did also some extra tests:

115,932,011 bytes, 479.973 sec., 4.590 sec., 7-Zip - -mx9 -md=1024m
116,165,709 bytes, 621.017 sec., 39.557 sec., NanoZip - -cO -m1g
119,482,502 bytes, 913.710 sec., 895.763 sec., NanoZip - -cc -m1g
134,069,478 bytes, 583.703 sec., 44.954 sec., NanoZip - -cO

[ScottX]

RZ SFX 1.0.1, changes:
- SFX module rewritten in VC++ 6.0 so no additional libraries neded (old module needed Visual C++ 2008 redist installed).
- RzSfxCreator - added progressbar and is now "Cancelable".
- both are now x86 apps

How to use it:

• Extract included ZIP to a directory.
• Copy Christian's RZ.exe to the same directory.
  I recomend to compress RZ.exe using UPX (because RZ.exe will be included to every self-extraciting EXE).
• Run RzSfxCreator.exe and create SFXs.

[Sportman]

Input:
481,136,694 bytes - ESA Gaia GDR1 Sky Map image as BMP

Output:
63,294,310 bytes, 275.040 sec., 267.395 sec., NanoZip - c
63,335,536 bytes, 690.925 sec., 695.219 sec., Zpaq - 510
63,397,150 bytes, 60.433 sec., 62.978 sec., MCM - x
63,948,375 bytes, 69.098 sec., 67.136 sec., ZCM - 8
64,840,936 bytes, 53.299 sec., 12.547 sec., NanoZip - O
65,150,080 bytes, 9.271 sec., 18.655 sec., Bsc - 6
67,618,893 bytes, 250.966 sec., 5.094 sec., Bzip2 - 9
68,154,202 bytes, 684.151 sec., 3.233 sec., RAZOR - -d 1023M
68,159,984 bytes, 613.183 sec., 3.125 sec., RAZOR - default
69,219,973 bytes, 223.276 sec., 4.205 sec., Glza - default
72,905,382 bytes, 176.461 sec., 2.327 sec., 7-Zip - 9
73,046,483 bytes, 187.171 sec., 3.208 sec., Lzturbo - 49
76,048,423 bytes, 187.379 sec., 3.655 sec., FreeArc - 9
77,326,665 bytes, 30.184 sec., 4.444 sec., Lrzip - 9
78,110,493 bytes, 449.965 sec., 1.295 sec., Brotli - 11 --large_window=30
78,462,213 bytes, 401.224 sec., 0.785 sec., Brotli - 11
79,894,824 bytes, 165.111 sec., 0.717 sec., Zstd - 22
82,625,303 bytes, 159.174 sec., 1.333 sec., WinRAR - 5
95,825,937 bytes, 112.511 sec., 1.505 sec., Zip - 9
96,569,304 bytes, 145.225 sec., 1.428 sec., CLZ - default

[jimbow]

could you add powerarchiver PA format? in many opportunities it performs like Razor but with faster compression speed.

[Sportman]

could you add powerarchiver PA format?

PA, Optimize Strong, Extreme:

2,262,582,680 bytes, 1194 sec., 601 sec., chainstate
61,655,558 bytes, 34 sec., 25 sec., nowiki
143,163,454 bytes, 587 sec., 42 sec., mongo
16,417,450 bytes, 35 sec., 28 sec., iis
71,540,886 bytes, 180 sec., 16 sec., gaia

[diskzip]

PA, Optimize Strong, Extreme:

2,262,582,680 bytes, 1194 sec., 601 sec., chainstate
61,655,558 bytes, 34 sec., 25 sec., nowiki
143,163,454 bytes, 587 sec., 42 sec., mongo
16,417,450 bytes, 35 sec., 28 sec., iis
71,540,886 bytes, 180 sec., 16 sec., gaia

Seems Razor is still the reigning leader over PA.