LZPM's future

[LovePimple]

I'm looking forward to the first release of this PowerHouse!

[Black_Fox1]

Black_Fox, dont keep silence - tell what you think about future lzpm!

OK As Im used to larger-dictionary packers, I have to ask, if 1MB dictionary isnt too small? On little files theres not much difference, but this is usually visible on my psd test file, which is bigger than usual files, but not i.e. tarred archives. On the other hand, winrar can score relatively good despite small dictionary...
Otherwise, I dont mind little ratio decrease on text files, if its weighted out with a bit larger ratio improvement

[encode]

if 1MB dictionary isnt too small?

With my current implementation, LZ with 1MB is already very slow, 2MB becomes unacceptable slower, leading just tiny compression gain in most cases.

[encode]

OK, the RZM defines a new level of LZ coders... Anyway, my LZ77 is still alive. Some testing results of a current version.

dictionary size: compressed size, compression time

valley.cmb

64K: 9,092,191 bytes, 8 sec
128K: 9,057,033 bytes, 13 sec
256K: 9,024,111 bytes, 23 sec
512K: 8,992,623 bytes, 41 sec
1M: 8,960,040 bytes, 77 sec
2M: 8,822,959 bytes, 162 sec

world95.txt

64K: 740,505 bytes, 1 sec
128K: 696,532 bytes, 3 sec
256K: 662,093 bytes, 6 sec
512K: 633,617 bytes, 13 sec
1M: 611,397 bytes, 43 sec
2M: 597,464 bytes, 125 sec

acrord32.exe

64K: 1,453,684 bytes, 1 sec
128K: 1,441,323 bytes, 2 sec
256K: 1,430,293 bytes, 4 sec
512K: 1,420,137 bytes, 8 sec
1M: 1,413,440 bytes, 16 sec
2M: 1,408,929 bytes, 32 sec

[Nania Francesco]

When it will go out the new release? seem to promise so much indeed!

[encode]

When it will go out the new release?

I just cant decide - what dictionary size to use? Also, what context for literals to use (the LC parameter in terms of LZMA)? Currently I keep 5-bit context. 4-bit one may be better for binary files like valley.cmb 8-bit is better for text files and already compressed files like A10.jpg. 5-bits should be the best in average.
Like you see with larger dictionaries, and especially on text files new LZ77 becomes much slower (just look at wolrd95.txt timings!). Anyway, 1MB is also should be the best at some point.
Check out results on pht.psd:

64K: 4,664,242 bytes, <1 sec
128K: 4,670,488 bytes, 1 sec
256K: 4,665,568 bytes, 2 sec
512K: 4,652,006 bytes, 3 sec
1M: 2,314,232 bytes, 6 sec
2M: 2,295,164 bytes, 14 sec
4M: 1,166,111 bytes, 20 sec

Overall, its a new QUANTUM-style encoder - LZ77 + arithmetic coding and a Storer&Szymanski parsing. Also for parsing I use constants plus a special cost function for offsets. We operate on a 16 MB blocks. During compression, the compressor shows progress:
<div class=""jscript""><pre>
optimizing 16384k block...
12%
[/code]

i.e. you see percentage...

Anyway, the release is soon - I just do an additional tests, tunings and improvements...

By the way, the new compressors name is BALZ...

[Nania Francesco]

Good work! Hi!

[encode]

But the main thing is a fast decompression and a very simple decoder. Decompression speed is even faster than LZPM's one. Of course it is faster than RZM's too - new LZ77 has about 2-4 times faster decompression than RZM...

[Black_Fox1]

1M: 2,314,232 bytes, 6 sec
2M: 2,295,164 bytes, 14 sec
4M: 1,166,111 bytes, 20 sec

This is quite surprising for me, my guesses are around 3 MB to spot that images are identical... that seems actually true but how come that 1MB dictionary already gains 30% ratio?

[encode]

It is also surprise for me - maybe with 1 MB we may find far more matches due to picture structure - i.e. beginning and ending of each image has repeated patterns. Like you see, 2 MB is not enough to find identical images, the gap between two images is larger or just slightly larger than 2 MB...