RZM - a dull ROLZ compression engine

[Nania Francesco]

RZM Vs 7ZIP [LZMA]
7ZIP 4.57 (Opt. -mx=9 - max 300 MB memory)
24.864.490 Comp. 120,073 Seconds - Dec 4,400 Seconds.
RZM v.0.06 (max 160 MB )
24.684.208 Comp. 144,537 s. - Dec. 7,095 s.

[LovePimple]

Another quick test...

Test machine: Intel PIII (Coppermine) @750 MHz, 512 MB RAM, Windows 2000 Pro SP4

Test File: ENWIK9 (1,000,000,000 bytes)

Timed with AcuTimer v1.2


ENWIK9 > 212,870,452 bytes
Elapsed Time: 02:16:33.485 (8193.485 Seconds)

[Christian]

Hi there!
Thanks for the detailed tests (and for hosting the files, Lovepimple)! Here comes one more. Ive done some minor tweaks - this gives a slight improvement in overall compression. Your millage may vary depending on the input-data.

Download 0.06b

Christian
i wonder if you will make ppm coder someday

Maybe, but for now Ive no plans doing it.

[Christian]

To improve of few the speed of decompression I recommend you not to visualize the statistics during the operations (it also of 5-10%)

Thanks for the clue. Ill give it a try, maybe it helps. But I visualize the status only every 64k or 128k - I dont remember.

[vcore]

quick test :

machine: Intel Q6600 2.4Ghz , DDR800 2GB Ram , Windows Vista

use AcuTimer v1.2 Copyright (c) 2007 by LovePimple


RZM 0.06b (Mar 3 2008-64M-64K), copyright (c) C. Martelock

Encoding 'enwik8' (using 258M memory)

97656kb -> 23864kb (24437254b, 24.44%), done.


Elapsed Time: 00 00:02:16.775 (136.775 Seconds)

final size ---> 24,437,261 bytes

[LovePimple]

Here comes one more. Ive done some minor tweaks - this gives a slight improvement in overall compression. Your millage may vary depending on the input-data.

Download 0.06b

Thanks Chris!

Mirror: Download

[Nania Francesco]

SFC Test-> 11.453.693 bytes Comp. 43,327 s. - Dec. 3,812 s.

C:>acutimer rzm c enwik8 t.a
AcuTimer v1.2
Copyright (c) 2007 by LovePimple
RZM 0.06b (Mar 3 2008-64M-64K), copyright (c) C. Mar
Encoding 'enwik8' (using 258M memory)
97656kb -> 23864kb (24437254b, 24.44%), done.
Elapsed Time: 00 00:02:25.190 (145,190 Seconds)

C:>acutimer rzm d t.a test.txt
AcuTimer v1.2
Copyright (c) 2007 by LovePimple
RZM 0.06b (Mar 3 2008-64M-64K), copyright (c) C. Mar
Decoding 't.a' (using 130M memory)
23864kb -> 97656kb (100000000b, 409.21%), done.
Elapsed Time: 00 00:00:06.890 (6,890 Seconds)

[Nania Francesco]

I have tried, for game, to compress MOC test with RZM adding a program (based on Rings 1.1 filters ) pre-filter for the images BMP,TIFF..WAVE. etc
My result has been -> 147.534.234 bytes (time 269,763 sec.) ! Very Very Very nice!

[LovePimple]

I have tried, for game, to compress MOC test with RZM adding a program (based on Rings 1.1 filters ) pre-filter for the images BMP,TIFF..WAVE. etc
My result has been -> 147.534.234 bytes (time 269,763 sec.) ! Very Very Very nice!

[Christian]

I have tried, for game, to compress MOC test with RZM adding a program (based on Rings 1.1 filters ) pre-filter for the images BMP,TIFF..WAVE. etc
My result has been -> 147.534.234 bytes (time 269,763 sec.) ! Very Very Very nice!

Thanks for the interesting test. Do you use delta coding (and channel seperation), or are the filters more advanced?

New day, new version. Only little adjustments.
RZM 0.06c

[Nania Francesco]

Delta coding (and channel seperation)! Hi!

[SvenBent]

Unreal Tournament 2004 CD6 in MDF/MDS format
then ran it through ECM, Precomp and rep


filename/method - size -decomp time
Originale - 615.577.164 bytes - 0s
7-zip ultra 256mb 273 - 584.899.521 bytes - 54s
RZM 0.6 - 580.128.828 bytes - 121s

So RZM beats 7-zip in compression, but decompression is more than twice the time of 7-zip

[Nania Francesco]

RZM 0,06C
SFC test
11.431.947 b. COMP. 42,797 s. - DEC. 3,844 s.
enwik8 test
97656kb -> 23857kb (24429590b, 24.43%), done.
Elapsed Time: 00 00:02:24.224 (144,224 Seconds)
UCLC Pgm files test
5.657.858 b. COMP. 5,938 s. - DEC. 2,516 s.
CALGARY Corpus test
821.321 b. COMP. 2,704 s. - DEC. 0,389 s.

[Christian]

7-zip ultra 256mb 273 - 584.899.521 bytes - 54s
RZM 0.6 - 580.128.828 bytes - 121s

So RZM beats 7-zip in compression, but decompression is more than twice the time of 7-zip

Thanks for the detailed test, Sven. Yepp, decompression of RZM is slower - though, the difference gets less, the higher the compression ratio. Still, imo decompression speed is very practical. But even if compression is slightly higher, RZM is experimental, only a few days old and has no frozen syntax, ... So, if youre looking for an archiving tool to backup your CD collection, maybe you should go with 7-zip.

Delta coding (and channel seperation)! Hi!

So, maybe we can achieve even better results with advanced transforms.

[LovePimple]

New day, new version. Only little adjustments.
RZM 0.06c

Thanks Chris!

Mirror: Download

[Black_Fox1]

Thank you! RZM has been tested preliminarily on my usual testset:
12 367 319 - RZM 0.06c - EDIT: missed 0.06c originally
12 370 286 - RZM 0.06b
12 502 511 - RZM 0.02
11 982 518 - CCM 1.30a

[Hahobas]

Christian, great to have you back! I really like RZM. Can't wait until you add those extra filters and really make it shine. Do you pull some of the same "cache tricks" that you used in ccm in rzm? Also, do you plan on doing any more work on CCM, or are you getting ready to make some new compressor that combines ideas from both RZM and CCM?

Anyway, here are some tests:

Test System: AMD Athlon 64 3000+ (Venice, single core) with 2GIG Corsair DDR CL2 RAM

rzm006b->rzm.exe c enwik8 (24437261 Bytes) - 3 m 14 s -
rzm006b->rzm.exe d enwik8 - 0 m 11 s -
rzm006c->rzm.exe c enwik8 (24429597 Bytes) - 3 m 14 s - slightly smaller
rzm006c->rzm.exe d enwik8 - 0 m 11 s - No Change

rzm006b->rzm.exe c enwik9 (210722103 Bytes) - 32 m 20 s -
rzm006b->rzm.exe d enwik9 - 1 m 52 s -
rzm006c->rzm.exe c enwik9 (210719085 Bytes) - 32 m 19 s - slightly faster slightly smaller
rzm006c->rzm.exe d enwik9 - 1 m 35 s - decompression 17.8% faster!

[SvenBent]

So, if youre looking for an archiving tool to backup your CD collection, maybe you should go with 7-zip.

I always do a md5 comparison of the decompressed file.
And I keep the decompresser(s) within the final 7-zip container.
just like with CCM(x) and PackJPG, to avoid incompatibilities due to changed format/specs.


and yes the decompression time it very practical. especially when you consider the possibility of Multithreading decompressing by using batch rutines for several files.
that way the 6 CDs only take 50% (roghtly) more time on a quadcore cpu.

[Christian]

Cant wait until you add those extra filters and really make it shine.

Thanks to the great feedback, I seriously consider adding filters. But still, theyll only improve (sometimes hurt) compression on pure RGB, WAVE or similar data structures.
Maybe Ill add table detection, too - but Im not so sure if its worth it. Bulat, if you read, do you have any numbers regarding the compression gain expected from table detection?

Do you pull some of the same "cache tricks" that you used in ccm in rzm?

Of course, Im using some tricks Ive learned during the development of CCM. But frankly, since RZM is LZ-based, it doesnt have much in common with CCMs architecture.

Also, do you plan on doing any more work on CCM

I still want to add PreComp to CCM. Besides that, maybe. The list with ideas and improvements to CCM got rather extensive - at some point I lost focus/interest.

getting ready to make some new compressor that combines ideas from both RZM and CCM

I thought about that. But Ive not made up my mind yet. If you combine a very strong symbol-coder with LZ, you end up using the symbol-coder most of the time - which makes the LZ-part useless. Therefore, imo the symbolcoder should only try to model things, the LZ-part cant.

[Christian]

I always do a md5 comparison of the decompressed file.
And I keep the decompresser(s) within the final 7-zip container.
just like with CCM(x) and PackJPG, to avoid incompatibilities due to changed format/specs.

You are doing your backups very thoroughly. In that case, you have one more choice with RZM.

that way the 6 CDs only take 50% (roghtly) more time on a quadcore cpu.

Very true. This must be one of the scenarios, in which a quadcore is of some real value (as long as IO isnt the bottleneck).

[Christian]

Thank you! RZM has been tested preliminarily on my usual testset:
12 367 319 - RZM 0.06c - EDIT: missed 0.06c originally
12 370 286 - RZM 0.06b
12 502 511 - RZM 0.02
11 982 518 - CCM 1.30a

Thanks a lot for the benchmark, BlackFox! Looking at those numbers, the syntax improvement really payed off.

[Black_Fox1]

If the BMP filter will have similar effectivity as that one in CCM, RZM will have better ratio than CCM!

[Christian]

I don't remember CCM's size on your benchmark without any filtering (iirc, 1.1.2a was the last one without filtering). If CCM started around 850.000 and achieved ~745.000 with filtering applied, than RZM will most probable gain only 80-120k, too.

[Bulat Ziganshin]

Bulat, if you read, do you have any numbers regarding the compression gain expected from table detection?

2% for very average binary file, 1mb for MFC, take into account that these numbers are for very sophisticated preprocessor, found only in rar&fa

you can test it youself, applying only table preprocessor to the data:
arc a a file -m=table --nodir

[Christian]

Thanks Bulat! Ill try it. So, I think Ill add table detection then, too. But before next weekend theres no time I can spare on RZM.

From the FreeArc thread:

[External compressor:rzm]
cmem = 256
dmem = 64
packcmd = rzm c $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
unpackcmd = rzm d $$arcpackedfile$$.tmp $$arcdatafile$$.tmp

im not sure about mem settings..

RZM uses ~258MB/~130MB for compression/decompression. The sliding Window is 64MB, but during decompression ROLZ needs additional 64M (256*64K*4) for the offsets, too.

[Black_Fox1]

I dont remember CCMs size on your benchmark without any filtering (iirc, 1.1.2a was the last one without filtering). If CCM started around 850.000 and achieved ~745.000 with filtering applied, than RZM will most probable gain only 80-120k, too.

Youre right, I have been mistaken a bit - CCM 1.1.1a scores ~950k, then there are two more versions inbetween featuring only small hashing and modelling tweaks, then comes 1.1.2a with ~815k, which still doesnt use any filters It seems I have never tested 1.15a, but it scores about 1000 bytes smaller than 1.1.2a... So it may be a bit less for mm filters. Total ratio is nevertheless brilliant for an assymetric coder, when compared to CCM.

[Bulat Ziganshin]

arc a a file -m=table --nodir

arc a a file -m=delta --nodir

So, I think Ill add table detection then, too.

unless you are going to use my code, your improvement will be smaller. ive spent several weeks on this engine

[encode]

Parsing is not much different from LZ77 parsing. I do forward parsing - it makes state propagation and string searching easier. I dont know what you want to know exactly?

As I dont want to reinvent the wheel, Ive just searched and found a nice post by Bulat. It describes the ideas of OP very well. For ROLZ it isnt any different - only the matches come from a different source.
Click me

One thing, you say that you use forward parsing - and in Bulat description we use backwards parsing.
In short, SS parsing can be defined as follows:
1. Find longest matches at each block position
2. Parse backwards, keeping sum of bits needed to encode each blocks position from end. We check each position, and if we find that better to encode literal sequence here, we drop the match.

Can you describe yours?

[osmanturan]

Can you describe yours?

Ill be very happy, too

[Christian]

One thing, you say that you use forward parsing - and in Bulat description we use backwards parsing.
In short, SS parsing can be defined as follows:
...
Can you describe yours?

Just read the description of bit-optimal parsing again. The cost optimization is done forward in this description. Only the final path construction is done backwards.