lzpm v0.16 is here!

[encode]

Please welcome the strongest LZ77 of my own.

What's new/features:

• LZ77 with 8 MB dictionary
• Enhanced S&S parsing on 32 MB blocks - i.e. at each buffer position we find not just one but a few good matches for further path optimization
• New match finder: HC5 - Hash Chains with 3-4-5-byte hashing
• Enhanced LZ-output coding
• Two compression modes: Normal ("e") and Extreme ("ex") for compression maniacs

[m^2]

Please welcome the strongest LZ77 of my own.

What's new/features:

• LZ77 with 8 MB dictionary
• Enhanced S&S parsing on 32 MB blocks - i.e. at each buffer position we find not just one but a few good matches for further path optimization
• New match finder: HC5 - Hash Chains with 3-4-5-byte hashing
• Enhanced LZ-output coding
• Two compression modes: Normal ("e") and Extreme ("ex") for compression maniacs

Thanks.
Quick test on Shelwein's Bookstar:
11358322B.

For comparison:
FreeArc LZMA:max:BT4 10746537
FreeArc LZMA:max:HT4 11669215
rzm: 10313547
LZTurbo -59: 10958180

[LovePimple]

Please welcome the strongest LZ77 of my own.

What's new/features:

• LZ77 with 8 MB dictionary
• Enhanced S&S parsing on 32 MB blocks - i.e. at each buffer position we find not just one but a few good matches for further path optimization
• New match finder: HC5 - Hash Chains with 3-4-5-byte hashing
• Enhanced LZ-output coding
• Two compression modes: Normal ("e") and Extreme ("ex") for compression maniacs

Thanks Ilia!

Mirror: Download

[osmanturan]

Tested on Core 2 Quad 2.4 GHz, 4 GB RAM, Vista Ultimate x64

              LZPM 0.16 (-e)   LZPM 0.16 (-ex)   WinRAR 3.70 (Best)
-------------------------------------------------------------------
a10.jpg             836,552           836,552               842,534
AcroRd32.exe      1,395,428         1,394,599             1,376,734
english.dic         885,771           885,759             1,114,867
FlashMX.pdf       3,748,589         3,747,952             3,749,331
FP.pdf              828,820           785,296               665,170
MSO97.dll         1,828,045         1,827,441             1,853,287
ohs.doc             815,136           811,218               806,708
rafale.bmp        1,023,495         1,022,087               821,991
vcfiu.hlp           676,282           674,223               668,617
world95.txt         588,652           583,478               464,069
Total Size       12,626,770        12,568,605            12,363,308
Total Time      124.632 sec       (VERY LONG)            16.090 sec

I found even (-e) mode slow enough. But, (-ex) mode has nearly PAQ8 like speed performance I know, you left this option for "compression maniacs". But, the compression score is not very different from (-e) mode while being hundred times slower. This is not good for a LZ coder (As a note, CMM4 is twice faster than even -e mode while having ~2 MB better compression). I think, you should remove (-ex) mode and make a new fast mode (faster than -e mode). I didn't test 7-zip because it would be unfair.

[encode]

I know that the compression difference between "e" and "ex" is small, I kept "ex" as an uncompromised variant - to get the deepest search from hash chains. Anyway, even multi-level hash chains are not intended for optimal parsing and such large dictionaries.

[chornobyl]

tried to test on my usual 512K binary file and failed, lzpm apeeared to be too memory hungry for my 512M of ram(384free) even in e mode. It wont run until i increase swap file to 512M and then with hdd thrashing tested on smaller file
64.raw 256x256 8bit raw image

unk        65536
lzmp016 e  51409 122s
lzmp016 ex 51409 118s
fpaq0f2    50999   1s
fpaq02     62373   2s

Havent you think about some f mode for faster compression and lower memory usage, while keeping e and ex modes as well.

[encode]

Yep, you need at least 1 GB RAM, which should be OK these days.

Don't forget - the catch is fast enough DEcompression.

"e" and "ex" modes use the same amount of memory with the same path optimization the only difference in hash chain length:

e - 256
ex - 16K



Well, this is an experimental release...

[Bulat Ziganshin]

for me, ex-like modes are good for testing strength of model, with the perfect match finder. so 1) you know how much you lose due to imperfectness of real matchfinder, 2) you may compare different coders ignoring their MF differences. so, it just targets other goal

[encode]

Just started working on the lzpm v0.17. New lzpm should be FAST! It is a complete rewrite of the compressor and it will have all new code parts and features:

• The FAST, newest and the most advanced LZP-based layer (32 MB dictionary)
• Extremely fast CM back end
• The standard E8E9 transformer

This is my first LZ-based compressor ever that will be fully optimized with my all new and special program optimizer. The BCM power mostly came from that optimizer!
The fastest LZ-layer came from my new and non-public fast compressor (the competitor to the THOR and others). I tried ALL possible LZ-driven ideas - LZ77, LZRW, ROLZ, LZP, ... done some theoretical conclusions about the algorithms, what leads to the best results in each case, what is the best feature of each modification, etc. And came up with this brilliant compression piece. After all, I decided to add some arithmetic, plus an optimized and simple enough modeler to get even more breathtaking results!
Additionally, I just realized that PAQ-speed compressors are out of business! We need fast, simple and yet efficient compression. By combining all my experience from LZ and CM areas (actual CM part is even further improved CM from the BCM!) we may get something very special!
(Drinking Vodka+Cola always helps!)

[osmanturan]

Do not drink to much else you can accidentally delete new LZPM's sources Anyway, I look forward to see new LZPM. If everything will be as you mentioned, my work-in-progress LZ codecs (a fast and slow variant) will have a new "competitor"

[LovePimple]

my work-in-progress LZ codecs (a fast and slow variant) will have a new "competitor"

That should be interesting!

[Fu Siyuan]

LZP?does it mean slower decompression speed than your other LZ coders? i am waiting for your new program! i also want to be your competitor.....but so busy in GRE and final exam these days .....

[encode]

Do not drink to much else you can accidentally delete new LZPM's sources

It doesn't matter - the entire source is in my head! That fact is the most important!

Anyway, I look forward to see new LZPM. If everything will be as you mentioned, my work-in-progress LZ codecs (a fast and slow variant) will have a new "competitor"

New LZPM has no competition! I'm just not really sure about that here are too many people that are experienced enough and have such programming skills to catch the uncatchable Joe!

LZP?does it mean slower decompression speed than your other LZ coders?

Not really, the decompression speed is fast enough - just slightly slower than the latest and final BALZ - it's due to the more heavier CM backend. As I said, the catch is fast compression, not only the decompression.

Anyway, I'm continue the design - many stuff might be changed easily...


Ah, forgot about my Vodka+Pepsi

[encode]

a group of cowboys sitting in the bar. Suddenly someone dashes into the bar, runs around and dashes out.
New cowboy asks: "who is that?"
- "this is Uncatchable Joe, nobody ever managed to catch him!"
- "why did noone catch him?"
- "who would bother doing that?.."