CMM fast context mixing compressor

[LovePimple]

Thanks Chris! I'm looking forward to the release of v3.

Mirror: Download

[LovePimple]

Quick test...

Test machine: Intel PIII @750 MHz, 512 MB RAM, Windows 2000 Pro SP4

Timed with AcuTimer v1.2


A10.jpg > 828,811
AcroRd32.exe > 1,443,772
english.dic > 561,138
FlashMX.pdf > 3,684,378
FP.LOG > 595,996
MSO97.DLL > 1,803,922
ohs.doc > 811,049
rafale.bmp > 760,302
vcfiu.hlp > 642,296
world95.txt > 526,937

Total = 11,658,601 bytes


ENWIK8 > 22,549,060 bytes

Compression time for ENWIK8 = 00:14:46.464 (886.464 Seconds)

[Nania Francesco]

Intel Core Duo 2 E6600 - DDR2 (1000) 2GB

Confirm result of
SFC TEST:
A10.jpg > 828,811
AcroRd32.exe > 1,443,772
english.dic > 561,138
FlashMX.pdf > 3,684,378
MSO97.DLL > 1,803,922
ohs.doc > 811,049
FP.LOG > 595,996
rafale.bmp > 760,302
vcfiu.hlp > 642,296
world95.txt > 526,937

Total = 11,658,601 bytes

ENWIK8 22.549.060 - 788,7 Kb/s

[toffer]

Thanks! I would be interested in some results comparing the previous with this release on a testset like SFC (including some binary data, where the mentioned new method gains some compression) - maybe MOC? I'm kindly asking for a test. Greets

[Shelwien]

@toffer:
http://shelwien.googlepages.com/test200801111103.htm

[toffer]

On my computer the speed gap between the two versions is larger. Looks like the most improvement is gained on very redundant files (fp.log, rafael.bmp, a starting point for transforming the context scheme (the other thread) into a (smaller) state machine. In my yesterday experiments the performance impact was less critical when using a context of 3 to 4 bits, instead of 6.

Something i forgot to mention is, that when you seperate the history maps, you can assign diffrent adaption speeds, since less predictable states should adapt slower to avoid the noise generated by nonstationarity (like compressing random data with a simple nonstationary model, which expands it).

With a context of 3 bits, the speed was almost identical, since this cmm version is a slight rewrite of the previous one, grouping similar operations together and only using two function calls per coded character (the blank version, without the context sensitive history map ran at ~390kb/s vs 330kb/s with the old version).

[toffer]

Here another quick release. I finally managed to get a proper SSE, my previous implementations just contained a tricky bug (written a "," instead of ";"). Speed and compression are improved. I think i almost squeezed out everything on the CM side, next is to make LZP work.

http://freenet-homepage.de/toffer_86/cmm2-080113.7z

N8

[Matt Mahoney]

http://cs.fit.edu/~mmahoney/compression/text.html#2072

[LovePimple]

Thanks Chris!

Mirror: Download

[LovePimple]

Quick Test...

A10.jpg > 833,347
AcroRd32.exe > 1,413,287
english.dic > 550,229
FlashMX.pdf > 3,686,716
FP.LOG > 598,115
MSO97.DLL > 1,773,988
ohs.doc > 811,388
rafale.bmp > 762,993
vcfiu.hlp > 624,700
world95.txt > 521,980

Total = 11,576,743 bytes

[toffer]

Thanks a lot!

[Shelwien]

http://shelwien.googlepages.com/cmm2_20080113.htm

[LovePimple]

Precomp v0.3.7 + CMM2-080113

A10.jpg > 700,202 *
AcroRd32.exe > 1,413,287
english.dic > 550,229
FlashMX.pdf > 2,589,078 *
FP.LOG > 598,115
MSO97.DLL > 1,773,988
ohs.doc > 710,510 *
rafale.bmp > 762,993
vcfiu.hlp > 624,700
world95.txt > 521,980

Total = 10,245,082 bytes

[toffer]

Hello everyone!

Here is a small cmm3 preview. It lacks most improvements i'm thinking of, at the moment (exspecially lzp). The main compression gain is caused by a simple match model (see my other post), a rewritten hash mapping and some minor changes.

http://freenet-homepage.de/toffer_86...080129_test.7z

[LovePimple]

Thanks Chris!

[LovePimple]

Hello everyone!

Here is a small cmm3 preview. It lacks most improvements im thinking of, at the moment (exspecially lzp). The main compression gain is caused by a simple match model (see my other post), a rewritten hash mapping and some minor changes.

http://freenet-homepage.de/toffer_86...080129_test.7z

Quick test...

A10.jpg > 831,196
AcroRd32.exe > 1,378,188
english.dic > 512,159
FlashMX.pdf > 3,673,536
FP.LOG > 482,187
MSO97.DLL > 1,751,757
ohs.doc > 767,587
rafale.bmp > 762,422
vcfiu.hlp > 548,486
world95.txt > 483,089

Total = 11,190,607 bytes


ENWIK8 > 22,235,659 bytes

ENWIK9 >193,752,743 bytes

[Nania Francesco]

Thanks Toffer! Nice !

[toffer]

Hi!

Here's another testing release. It is compiled for around 400mb of RAM and adds some small tweaks (i tried the paq state machine instead of mine, which gave a slight compression gain normally, but hurts for less compressible files) - the main improvement is the optimized mmx code for mixing, which gives a speedup of around 15% on my computer. The small memory version is even faster - i could boost the speed from 320 kb/s to 400 kb/s (1.8GHz Sempron) - since it spends less time on cache misses (fewer RAM) and mixing is more significant.

http://freenet-homepage.de/toffer_86...080207_test.7z

I would be interested in a test on a large fileset, like Nania's MOC.

Thanks in advance

[Nania Francesco]

Hi Toffer! Updated in MOC!

[Nania Francesco]

SFC Test 11.123.559 bytes in 85 sec. (91 sec. decompression)

[LovePimple]

Thanks Chris!

[Matt Mahoney]

http://cs.fit.edu/~mmahoney/compression/text.html#1796

( Decompression time is probably high because I was watching a video while running cmm at low priority. I use timer 3.01 process time, but I guess it's not perfect )

[LovePimple]

Quick test...

A10.jpg > 834,990
AcroRd32.exe > 1,368,984
english.dic > 499,477
FlashMX.pdf > 3,673,346
FP.LOG > 468,700
MSO97.DLL > 1,735,260
ohs.doc > 766,537
rafale.bmp > 757,139
vcfiu.hlp > 540,661
world95.txt > 478,079

Total = 11,123,173 bytes

[LovePimple]

Another quick test...

Test machine: Intel PIII (Coppermine) @750 MHz, 512 MB RAM, Windows 2000 Pro SP4

Test File: ENWIK8 (1,000,000,000 bytes)

Timed with AcuTimer v1.2


ENWIK8 > 21,212,766 bytes
out/in: 20715/97656, 21.21% (1.70 pbc); speed: 120.1 kB/s
Elapsed Time: 00:13:33.115 (813.115 Seconds)

[toffer]

Thanks!

[toffer]

Good evening!

I've completely changed my mixing architecture to NN's and added a simple mixing hierarchy. There's a match model, which uses a 16 mb sliding window. I use a simple heuristic to turn models off, when the higher order models seem to offer good predictions. The source has almost completely been rewritten. In the near future I'll add a compression option switch, which controls the memory allocated.

Here's a testing release of cmm4:

http://freenet-homepage.de/toffer_86/cmm4.exe

And now there's a version number

[LovePimple]

Awesome! Thanks toffer!

[LovePimple]

Quick test...

A10.jpg > 831,088
AcroRd32.exe > 1,337,313
english.dic > 503,406
FlashMX.pdf > 3,659,660
FP.LOG > 485,520
MSO97.DLL > 1,701,679
ohs.doc > 759,871
rafale.bmp > 746,667
vcfiu.hlp > 530,022
world95.txt > 476,505

Total = 11,031,731 bytes

[LovePimple]

Another quick test...

Test machine: Intel PIII (Coppermine) @750 MHz, 512 MB RAM, Windows 2000 Pro SP4

Test File: ENWIK8 (1,000,000,000 bytes)

Timed with AcuTimer v1.2


ENWIK8 > 21,459,665 bytes
Elapsed Time: 00:10:56.384 (656.384 Seconds)

[toffer]

Thanks!

cmm3 080207 - 813.115 Seconds
cmm4 v0.0 - 656.384 Seconds

-> Around 20% faster without any ASM

BTW: I tested this against a ripped off version of lpaq (static counters, no second SSE stage and text model - which equals my layout). Results are almost equal, only 10k difference, while beeing almost twice as fast (at least on my machine).