RAZOR - strong LZ-based archiver

[Samantha]

A quick feature request (besides ECM prefilters). would it be possible to make it so if you just drag and drop a .rz archieve unto rz.exe it will automatic decompress it (create folder with the name of the archieve).
It would help a lot for people that are not command line savy.

I do not understand what the problem or the difficulty to use it.......I think this magnificent compressor should only be used as a command line to be implemented in various solutions, Who is not capable of giving such a command [ rz x -y -o "C:\New Folder\Output" "C:\New Folder\enwik8.rz" * ] I think it is not worthy to use it.

[redrabbit]

Hi everyone,

I want to share with you my latest archiver-project called "RAZOR".

Silently, I've been working on compression-related things. But I need to close this whole chapter for now, although some core-features are not yet implemented. Nonetheless, I don't want RAZOR to get buried in my backups.

RAZOR is a highly asymmetrical LZ-based archiver. It's key properties are:
-strong compression ratio / fast decompression speed
-moderate memory requirements for decompression (1.66N)
-deduplication / special treatment for executables & special data types
-slow compression / high memory footprint for compression

Please report bugs (if you find any) to my e-mail address. Please do not ask for new features. This is a test version - so at least verify your archives.

Have fun!

Christian

Just AMAZING!! i started to compress some games wich are using Unityengine
As example
Hollow Knight (8,87 gb's)
srep + 7z (1,15 gb's)
razor with dictionary 512 (1,07 Gb's)

I tooks 7-8 hours to compress with Razor 1.00, but the decompression is more more faster

It is possible to have a unpacker binary x86 for windows and Linux? or btw.. do you will open the code?

[redrabbit]

Well after some tests (4) in some games RAZOR 1.00 saves between 60-200 mb's (i used "small" games -not more than 10gb's-)

[Dimitri]

excellent Tool, But can you provide a stdio exe??

[Lucas]

Quick test of RAZOR on enwik8 with dictionary preprocessing:

rz a -d 100m
21,992,522 (no preprocessing)
21,459,771 (wbpe preprocessing)
20,899,822 (xwrt preprocessing)

For comparison bsc v3.1.0 -b100 compresses to 20,920,018.
From the looks of it if you add a basic dictionary coder RAZOR will be pretty much on par with bwt compressors on text without effecting decode speed too much.

Great job Christian!

[Skymmer]

1.) A small bug report (actually cosmetic one)
During decompression the progress value in title bar (as well as in taskbar) never reaches 100% value. More exactly speaking when decompression finished, the console window reports not the 100% value.
It is absolutely non-critical but anyway I decided to report this issue. See attached screenshots.
The system is (if its important): Windows 7 Pro x64 SP1 (6.1.7601)



2.) LTCB entry test
Compression results and other measurements for both enwik8 and enwik9 are below.
Tested on: i7-2700K@4700MHz, clean environment, ProcProfile v1.5.1 as timing utility
Razor Options: -d 1023M (Anyway Razor limits window size to file size if its smaller)

ENWIK8

enwik8.rz: 21 992 522 bytes

compression
---------------------------------------
Process Time     : 237.027s
Clock Time       : 177.059s
Working Set      : 1157 MB
Pagefile         : 1250 MB
IO Read          : 95 MB (in 1527 reads)
IO Write         : 20 MB (in 208 writes)

decompression
---------------------------------------
Process Time     : 1.278s
Clock Time       : 1.287s
Working Set      : 154 MB
Pagefile         : 163 MB
IO Read          : 20 MB (in 208 reads)
IO Write         : 95 MB (in 1526 writes)

ENWIK9

enwik9.rz: 173 041 176 bytes

compression
---------------------------------------
Process Time : 3341.728s
Clock Time   : 2529.583s
Working Set  : 10118 MB
Pagefile     : 11262 MB
IO Read      : 953 MB (in 15260 reads)
IO Write     : 165 MB (in 1471 writes)

decompression
---------------------------------------
Process Time : 11.746s
Clock Time   : 11.762s
Working Set  : 1343 MB
Pagefile     : 1572 MB
IO Read      : 165 MB (in 1471 reads)
IO Write     : 953 MB (in 15259 writes)

And here comes the tricky part. According to rules of LTCB the decompressor must be provided either as zipped sources either as zipped executable decompressor.
The rz.exe by itself can be compressed to 89 117 bytes (kzip /b512, Apr 14 2007) but this result can be improved.
We can compress rz.exe with UPX. Then we can clean the DOS stub message, sections names and UPX signatures and resulting ZIP size is 78 917 bytes (kzip /b128, Apr 14 2007) but I don't know if packing the executable violates the terms of use which are stated in RAZOR:

DO NOT directly or indirectly modify, decompile or otherwise reverse engineer this software.

Also, we can improve the result further. Simply by cleaning the GCC compiler strings and runtime warning\errors, then packing the EXE with UPX, cleaning the DOS stub message, sections names and UPX signatures and resulting ZIP size is 78 411 bytes (kzip /b256, Apr 14 2007). Nice. But I think its the violation.
So, in order to not provide any harm I will not post the modified executables here until such way of size improvement will be approved by author.
Anyway, the UPX settings:

--all-filters --force --small --lzma --best --compress-exports=1 --strip-relocs=1 --crp-lzma-ds=8388608 --crp-lzma-fb=273 --crp-lzma-lc=3 --crp-lzma-lp=0 --crp-lzma-pb=0

So, the result of RAZOR for LTCB can be: 173 119 587 bytes
With such result RAZOR can take 32th place, replacing (what an irony) the CCMX.

[SvenBent]

just another comparison

Win98se Dan & Eng ISO  + IE6 & OE6 updates (92files 6 folders)

                                                 Dtime    Ctime
Uncompressed       1.21 GB (1,310,583,827 bytes)

7-zip Ultra/64MB   1.00 GB (1,083,770,976 bytes)  0:21     3:52
7-zip Ultra/768MB   696 MB (730,210,025 bytes)    0:13     5:08
M7repacker 768MB    696 MB (730,176,236 bytes)    0:14  5:54:06

WinRAR4 Best/4MB   1.02 GB (1,105,137,093 bytes)  0:11     0:37
WinRAR5 Best/64MB  1.01 GB (1,090,760,757 bytes)  0:07     1:00
WinRAR5 Best/1GB    704 MB (738,888,786 bytes)    0:04     0:57

Razor 1MB           705 MB (739,819,680 bytes)    0:15    11:20
Razor 4MB           697 MB (731,499,384 bytes)    0:14    11:57
Razor 16MB          692 MB (726,416,614 bytes)	  0:16    12:27
Razor 64MB          691 MB (725,107,990 bytes)    0:17    14:00
Razor 256MB         689 MB (723,496,654 bytes)    0:15    13:54
Razor 768MB         688 MB (722,074,583 bytes)    0:18    14:19

- Razor deduplication really shows its strength on this. The 2 iso are almost identical and razor takes advantages of that already at 1MB dictionary.
- decompress time is not that far off 7zip.
- Compresison time is an a tad slower ( but accetpable in my opinion)
- Rar5 has a really fast decompression
- Did a few SHa1 hash verifiation of compression/decompression on razor and all came out fine.

[SvenBent]

I do not understand what the problem or the difficulty to use it.......I think this magnificent compressor should only be used as a command line to be implemented in various solutions, Who is not capable of giving such a command [ rz x -y -o "C:\New Folder\Output" "C:\New Folder\enwik8.rz" * ] I think it is not worthy to use it.

Well you are definatly not without a good point. But sadly a lot of my friends has never touched command line. I literrally had one ask me why i still worked in command line. And had to explain him the control benefits/ scripting benefits.

[jibz]

We can compress rz.exe with UPX. Then we can clean the DOS stub message, sections names and UPX signatures and resulting ZIP size is 78 917 bytes (kzip /b128, Apr 14 2007) but I don't know if packing the executable violates the terms of use which are stated in RAZOR

Just mentioning that, from my understanding as a non-lawyer, the UPX license restricts you from making any changes to the compressed executable that makes it unable to be decompressed with UPX again. You can only do that if the program being compressed is also distributed under the terms of the GPL.

[SvenBent]

Some more bennchmark razor really takes the crown her.

Civilization II Gold  Nero Raw CD image with audio tracks + Updates (7 files 1 folder)

                                                    Dtime    Ctime
Uncompressed            718 MB (753,027,898 bytes)
ECM                     660 MB (693,035,692 bytes)   0:07     0:26

7zip Ultra/64MB         544 MB (571,141,165 bytes)   0:24     2:35
7zip Ultra/256MB        543 MB (569,874,099 bytes)   0:24     3:01
7zip Ultra/1GB          543 MB (569,864,179 bytes)   0:24     2:30
M7zRepacker 1GB         543 MB (569,874,099 bytes)   0:22  3:12:48

Rar4 Best/4MB           509 MB (534,008,619 bytes)   0:08     0:22
Rar5 Best/64MB          505 MB (530,146,752 bytes)   0:04     0:39
Rar5 Best/256MB         505 MB (529,904,586 bytes)   0:03     0:53
Rar5 Best/1GB           505 MB (529,861,029 bytes)   0:03     0:47

Razor 4MB               447 MB (468,974,395 bytes)   0:16    11:19
Razor 16MB              443 MB (465,068,964 bytes)   0:17    11.51
Razor 64MB              441 MB (462,821,195 bytes)   0:16    12:24
Razor 256MB             441 MB (462,765,131 bytes    0:16    13:21
Razor 1023MB            441 MB (462,650,123 bytes)   0:15    13:22

ECM + 7zip Ultra/64MB   489 MB (513,662,167 bytes)   0:21     2:28
ECM + 7zip Ultra/256MB  489 MB (512,972,140 bytes)   0:21     2:45
ECM + 7zip Ultra/1GB    489 MB (512,799,832 bytes)   0:21     2:16
ECM + M7Repacker 1GB    450 MB (472,841,861 bytes)   0:20  2:48:15

ECM + Rar4 Best/4MB     453 MB (475,159,589 bytes)   0:07     0:20
ECM + Rar5 Best/64MB    450 MB (472,682,503 bytes)   0:02     0:36
ECM + Rar5 Best/256MB   450 MB (472,436,189 bytes)   0:04     0:47
ECM + Rar5 Best/1GB     450 MB (472,332,344 bytes)   0:04     0:47


ECM + Razor 4MB  	392 MB (411,781,819 bytes)  00:13    10:26
ECM + Razor 16MB        388 MB (407,517,734 bytes)  00:11    11:38
ECM + Razor 64MB        387 MB (406,001,189 bytes)  00:14    11:38
ECM + Razor 256MB       387 MB (405,819,136 bytes)  00:14    12:08
ECM + Razor 1023MB      387 MB (405,804,726 bytes)  00:14    12:17

- Still looks like 64mb was not a bad choice for default dictionary size
- also this is a CD image with about 260mb audiotracsk. and razor really seems to shine on this
- ECM prefiltering actually made he compression a bit faster but the decompression a few secs slower. but gained a pretty amount of size reduction

[Shelwien]

Try http://freearc.dreamhosters.com/mm11.zip with 7z maybe?
Also -myx=9?

[SvenBent]

Diable 2 + Lord of destruction ( 2iso images and 2 cue/bin image prefilter with ECM)

M7zrepacker  1.81 GB (1,951,629,320 bytes)
Razor 1023mb 1.70 GB (1,828,264,524 bytes)

[SvenBent]

Beowulf DVD Ahlcohol image + patch (3 files 1 folder)

Original         5.07 GB (5,445,782,056 bytes)
M7repacker 1GB   4.70 GB (5,046,846,048 bytes)
Razor 1203MB     4.69 GB (5,045,971,255 bytes)

- Pretty much even but the data did not compress well at all

[Christian]

SvenBent, thank you for all the tests. I'm happy to see, that rz does not only work good on my private test-sets. If data is already compressed, there's not much difference between rz, 7zip, rar, ... This is precomp/reflate territory.

excellent Tool, But can you provide a stdio exe??

rz supports unicode by using the windows-api. Therefore, no stdio/linux compile.

Quick test of RAZOR on enwik8 with dictionary preprocessing:

rz a -d 100m
21,992,522 (no preprocessing)
21,459,771 (wbpe preprocessing)
20,899,822 (xwrt preprocessing)

For comparison bsc v3.1.0 -b100 compresses to 20,920,018.
From the looks of it if you add a basic dictionary coder RAZOR will be pretty much on par with bwt compressors on text without effecting decode speed too much.

Great job Christian!

Thanks Lucas. I evaluated this, too.

1.) A small bug report (actually cosmetic one)
During decompression the progress value in title bar (as well as in taskbar) never reaches 100% value. More exactly speaking when decompression finished, the console window reports not the 100% value.
...
So, the result of RAZOR for LTCB can be: 173 119 587 bytes
With such result RAZOR can take 32th place, replacing (what an irony) the CCMX.

The windows-API documentation for SetConsoleTitle: "When the process terminates, the system restores the original console title."
But the title is not restored. I've added a new build to the first post, which fixes the ConsoleTitle bug. This build is using PGO.

About ENWIK9: Thank you for the results. This is news to me - I do have only 8GB memory. I was hoping for a better result. No need for UPX - the rank stays the same. Btw, I like this irony.

---
Is it possible to edit the first post and add the new build? Thank you, Ilia!

[avitar]

Hi Christian, can you pls supply a win32 version, in the zip. TIA John

[encode]

Is it possible to edit the first post and add the new build?

Now it's possible! But if I'll see vandalism acts on this forum once again, I'll disable this feature forever!

[zyzzle]

I would like very much to try RAZOR on my x86 32-bit machine. May we please get a 32-bit compile? This looks fabulous so far, I'd like to see how much of a hit compression / decompression time takes on a 32-bit system.

[SvenBent]

I would like very much to try RAZOR on my x86 32-bit machine. May we please get a 32-bit compile? This looks fabulous so far, I'd like to see how much of a hit compression / decompression time takes on a 32-bit system.

I apologize if my question comes of rude but I'm quite the curios person.
Why 32bit when we have had 64bits for almost 2 decades now ?

[avitar]

Not everyone has 100% 64 bit pcs and not everyone is compressing 10s or 100s of Gbyte. I have 1 or 2 that I use for data collection. I'd like to compare razor on relatively small files with ccm/ccmx, arc, 7z all of which are available in 32 bit (btw for my xml/ascii data ccmx is best I've found!). There seem to be quite a few people asking for 32 bit version - guess it isn't much work for christian to generate it. So why not? j

[Samantha]

I apologize if my question comes of rude but I'm quite the curios person.
Why 32bit when we have had 64bits for almost 2 decades now ?

Because for another decade there will be people who will still use 32 bit systems...

[SvenBent]

C&C Tiberiean Sun + Firestorm CloneCD image (ECM prefiltered) 10 files

Original    2.07 GB (2,232,732,713 bytes)
M7repacker  1.18 GB (1,268,279,237 bytes)
Razor       1.15 GB (1,235,737,055 bytes)

Razor still edging out a win compared to bruteforcing 7zip.

[SvenBent]

LotR - The battle for Middle-earth 2 iso
 
Original               5.53 GB (5,946,890,264 bytes)
Winrar                 5.42 GB (5,823,590,606 bytes)
7zip ultra/1.5GB/273   5.39 GB (5,789,727,484 bytes)
Razor                  5.38 GB (5,779,359,376 bytes)

- Razor still edging out a win in hard to compress cases
- Its nice to see how razor manges to outperform both 7zip and Winrar at the same time on everything ive tested so far.
I used both and keep the smallest file due to them changing places at beeing the best depending on data type.
m7repacker fixed most of that but came at an enormous cost in encoding time.
Razor arrived and is a true champion and finnally a userable solution

[SvenBent]

Because for another decade there will be people who will still use 32 bit systems...

This does not in anyway or shape addressee the why of it though.

i probably should have dropped this but i cant help noticing you weird stands on your last two answers.

First you take en elitist stand that people not being able to work with command line effectively should not have the honor of using razor.
on the same logic I could argue by your logic that people with no 64bit capabilities should not have the honor of using razor as well.

But I didn't because instead of using forum post as a spring board to try to boost my own ego, I asked into the why of it to get an understanding of the issue at hand.
I don't know if avitar wanted to use it on some atom powers unit with just 32 bit or some specialty unknown (for me) embedded software system, of it was just a lack of easy access 64bit system on that part of the world, or some 4th reason that i cant think off.
So instead of taking the holier than though attitude that you brought up earlier... I simply asked into the reason to gain some understanding of the why of it.

You see in my to spot of the world (Northen Europe and Southern US) I haven't meet a reason in my experience anything running 32bit anymore. Which is also why i was surprised when i was asked to make a 32bit version of my software ( which i did)
But now I know that one of my friends living in venzuella does not have the same daily luxuries as i can take for granted.

So at this point there was a possibility to "connect" with someone that might have a different world view and create and understanding from a different view point. I could get an understanding of the why of 32bit and the 32bit user could get more awareness of the issue.
But instead you answered with an absolute zero value post because it did in fact not even get close to addressing the why of it.


I have enjoyed encode.su for many years because it did not have the usual smart ass and ego boosting poster and and mostly criticism and suggestion was taken with the mind of a scientist and addressed logically.
I would hope this does not change and we don't get all these smart ass comments, where people just post to affirm themselves.
If that is the reason you are posting please fell free to not answer any of my question as the answers are useless for me anyway, and those kinds of response just makes people look less "mature" in my eyes.


Anyway my question is still open if any 32bit only users would like to address it. And if not that is fine as well. I was simply curious to the issue at hand.

[Samantha]

@SvenBent The concept is simple, and your long talk makes little sense, if anyone asks for a 32bit version, it's because it needs to try it, use it, test it in a 32bit system, I'm also interested in a 32bit version, despite I use 64bit systems, but I have friends and colleagues who still use 32bit systems, who wants to test it and create archives with RZ in their systems, they can not do it. Also a mature person would have replied that it was not possible to make a 32bit version, or that he would not have seen the need to do so. Lesson of the continents around the world I do not care, I have already visited them and already know them. Anyway, it's useless to warm you if you can not have a 32bit version, nothing happens, we both continue to live happy and satisfied for this version.

[SvenBent]

Overlord  alchohol dvd image

Original               4.16 GB (4,473,394,768 bytes)
Winrar5  1GB           2.23 GB (2,400,519,128 bytes)
7-zip Ultra 1.5G/277   2.05 GB (2,209,683,757 bytes)
7-zip + srep           1.97 GB (2,121,441,608 bytes)
Razor 1023MB           1.81 GB (1,946,256,794 bytes)

I finnaly found a test file with 7-zip benefiting from srep to compare against razor. Razor still taking the compression lead.

[SvenBent]

@SvenBent, if anyone asks for a 32bit version, it's because it needs to try it, use it, test it in a 32bit system

Still does not answer the why this need. If its plain curiosity etc etc. already explained this. This question is only "simple" because you don't answer it.

@SvenBentBut I have friends and colleagues who still use 32bit systems, who wants to test it and create archives with RZ in their systems, they can not do it.

So its a available factor that important. Again I find it funny that you argue for this when its in YOUR interest. Yet you take this holier than though attitude when its for others and not you. Even though the basic underlining argument is the same. Providing the ability to benefits from razor in a larger audience.

@SvenBent, Also a mature person would have replied that it was not possible to make a 32bit version, or that he would not have seen the need to do so.

I'm not sure what point you are going at here. I am rather confused if you think a mature person should have replied 32bit is not possible on software wheert he person making it has not even responded yet perhaps even seen the request yert. Or if i should have replied that on the software i did make into a 32bit version... its seems to defy logic on both account.
Can you please explains this?

@SvenBent, Lesson of the continents around the world I do not care, I have already visited them and already know them.

That is very good for you. However I don't which is exactly why i put out question to people to learn about there situations.

@SvenBentAnyway, it's useless to warm you if you can not have a 32bit version, nothing happens, we both continue to live happy and satisfied for this version.

again im not following your logical here. at no time have i denied 32bit for anyone or spoken against a 32bit version.
as i already mentioned i made 32bit version available of my software when requested so.

I simply in this thread just asked into the reason why people still where using 32bit systems, and i did it very polity iirc. Because sometimes when you ask about things you cant get some marvelous answers and obtain a new set of knowledge.
All in all curiosity is a huge drive within science.

[zyzzle]

My valid reasons for requesting a 32-bit compile of RAZOR:

1. Atom processor
2. Compare it against 32-bit 7zip at -md64 or -md128.
3. running x32 Windows and installing 64-bit not suiting my needs due to: a) no NTVDM support in 64-bit Windows. b) no fullscreen command prompt in x64 Windows. Granted these are "artificial" limitations because Microsoft is too lazy / greedy to implement them in x64 Windows.
4. Smaller files to compress, no need for 10s of GB.
5. Desire to stick with LZ-based archivers due to extremely fast decompression speed.

[avitar]

Some of you seem to making a big deal re 32 bits-surely point is if it can be done easily why not? j

[diskzip]

I think the more interesting question here would be if a DLL-based API could be provided at some future point.

I have tested this compressor and it performs wonders on the DiskZIP standard data-set - as some of you on this forum know, I have already been through Oodle (terrible customer service, deliberately non-testable product, no native archiving support), and PowerArchiver (worse compression ratio) in trying to find something that can outperform DiskZIP's 7-Zip plug-in at the super-high compression settings.

The only drawback to Razor so far is that it takes twice as long as DiskZIP to compress. However the space savings are really quite incredible - 2.12 GB compressed (Razor), vs. 2.47 GB compressed (DiskZIP).

I would love to implement a Razor plug-in for DiskZIP, on terms that its original author would allow, of course, if a DLL were available. I would also be ready on hand to help with the development and testing of that DLL, its callback functions, etc. In fact, it has been so long that I have seen a strong new compressor enter the market-place, that this work would give me tremendous joy. The last time I witnessed something like this was the birth of 7-Zip, well over a decade and half ago!

[Gonzalo]

As of today, I think WCX Totalcommander plugins are the de facto standard of data compression dlls. They have a well defined way of handling most if not all parameters somebody would need to include its functionality into a separate program. Various commander-style programs include support for those plugins, even other apps like PowerArchiver. So, if released as a WCX plugin, Razor becomes automatically available to an enormous amount of people _and_ gets a GUI for free. (Archives in those 'commander' apps are treated almost like a normal folder).

I also feel the same way about Razor. It would be a great replacement of LZMA on installers like those of NSIS, for example. I also think a non-archiver version that can be used as a codec could have a very promising future.

See this for an example of WCX uses:
https://totalcmd.net/directory/packer.html