LZP flag sequence compression

[Shelwien]

"pinc" on IRC posted a sample flag sequence from his LZP implementation.
And surprisingly paq8 shows much worse results than 7-zip for it,
so I got interested and experimented a little:

http://ctxmodel.net/files/mix_test/m...t_vA2-pinc.rar

Experiment with CM tuning for LZP control sequence

779280 pinc_sample 
 68753 paq8p -6
 62166 7z 9.04 -mx=9
 60924 7z ??? (reported by chornobyl)
 71329 mix_test 09h2x62
 59988 mix_test 09h2x6a

09h2x62 - single counter / static allocation
09h2x6a - 8 counters / dynamic allocation

(59988 is an intermediate result, current one is already 59930,
but I didn't bother to update it).

The mix_test result shows that its probably not a matter
of mixing vs LZ optimal parsing, as mixing still wins.
But then what's the matter with paq8?

[Matt Mahoney]

Maybe paq8 lacks an order-17 model.

C:\tmp\mix_test_vA2-pinc>ppmonstr e -o16 -fpinc_sample.pmm-o16 pinc_sample
Monstrous PPMII compressor based on PPMd var.J, Feb 16 2006
   pinc_sample: 779280 >  76587, 0.786 bpb, used:  2.5MB, speed: 686 KB/sec

C:\tmp\mix_test_vA2-pinc>ppmonstr e -o17 -fpinc_sample.pmm-o17 pinc_sample
Monstrous PPMII compressor based on PPMd var.J, Feb 16 2006
   pinc_sample: 779280 >  63122, 0.648 bpb, used: 24.4MB, speed: 551 KB/sec

I also looked at the file with fv, but the order-17 structure was not obvious.

[Darek]

Hi!
Iheve question - don't blame me if it's stupid: are you sure that output file from 09h2x6a is correct? Have you compared uncompressed files with original (CRC)?

I've tried to use this version on my testbed files and all scores are better than paqxxx compressed files and ... it's strange.

It looks like some part of files are missed in transformation or I'm wrong and this version of r1c1 file couldn't be used to other files than original book1bwt or pinc_sample...

Darek

[Shelwien]

Well, obviously its a flag model... it encodes only bit0 of each byte.
Nobody said that it can be used with other files and/or would be always better than paq8

[Darek]

Ok,
sorry for question.

[chornobyl]

best result so far

[Skymmer]

"pinc" on IRC posted a sample flag sequence from his LZP implementation.
And surprisingly paq8 shows much worse results than 7-zip for it,
so I got interested and experimented a little...

Realy interesting sample. I did a couple of tests with some other programms.
-6 level used for PAQs. 512MB\Fast Analysis for PWCM and 256MB\512 (Largest optimised match)\Fast Analysis for ROLZ.

paq8p           68 753
paq8p3          68 665
paq8px          69 013
paq8k           78 133
winrk_pwcm      73 555
winrk_rolz3     59 131

best result so far...

Damn! 60 468 bytes with 7z its realy good! I'm still trying to beat this result but probably you have used some tricks. At least one surely. I mean that you somehow managed to make the file inside archive to have no name
Anyway, my best result so far is 60 476 bytes.

[chornobyl]

If your result is 60476 than you probably win, because my version with name takes 60487, and to remove name i used 7zFM. Also xz variant is even smaller because of smaller headers.

[Skymmer]

If your result is 60476 than you probably win, because my version with name takes 60487, and to remove name i used 7zFM. Also xz variant is even smaller because of smaller headers.

First of all thanks for "rename" tip! It helped a little but I was able to reach 60 469 only which is 1 byte larger than yours. Damn! 1 byte again
Second, the probably term is not an option for me.
Third, after some additional bruteforcing I was able to produce 60 416 bytes version. By the way not with v9.04 and not even with v4.65