Page 3 of 6 FirstFirst 12345 ... LastLast
Results 61 to 90 of 172

Thread: CMM fast context mixing compressor

  1. #61
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Thanks Chris! I'm looking forward to the release of v3.

    Mirror: Download

  2. #62
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Quick test...

    Test machine: Intel PIII @750 MHz, 512 MB RAM, Windows 2000 Pro SP4

    Timed with AcuTimer v1.2


    A10.jpg > 828,811
    AcroRd32.exe > 1,443,772
    english.dic > 561,138
    FlashMX.pdf > 3,684,378
    FP.LOG > 595,996
    MSO97.DLL > 1,803,922
    ohs.doc > 811,049
    rafale.bmp > 760,302
    vcfiu.hlp > 642,296
    world95.txt > 526,937

    Total = 11,658,601 bytes


    ENWIK8 > 22,549,060 bytes

    Compression time for ENWIK8 = 00:14:46.464 (886.464 Seconds)

  3. #63
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    222
    Thanked 146 Times in 83 Posts
    Intel Core Duo 2 E6600 - DDR2 (1000) 2GB

    Confirm result of
    SFC TEST:
    A10.jpg > 828,811
    AcroRd32.exe > 1,443,772
    english.dic > 561,138
    FlashMX.pdf > 3,684,378
    MSO97.DLL > 1,803,922
    ohs.doc > 811,049
    FP.LOG > 595,996
    rafale.bmp > 760,302
    vcfiu.hlp > 642,296
    world95.txt > 526,937

    Total = 11,658,601 bytes

    ENWIK8 22.549.060 - 788,7 Kb/s

  4. #64
    Programmer toffer's Avatar
    Join Date
    May 2008
    Location
    Erfurt, Germany
    Posts
    587
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Thanks! I would be interested in some results comparing the previous with this release on a testset like SFC (including some binary data, where the mentioned new method gains some compression) - maybe MOC? I'm kindly asking for a test. Greets
    M1, CMM and other resources - http://sites.google.com/site/toffer86/ or toffer.tk

  5. #65
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,912
    Thanks
    291
    Thanked 1,272 Times in 719 Posts

  6. #66
    Programmer toffer's Avatar
    Join Date
    May 2008
    Location
    Erfurt, Germany
    Posts
    587
    Thanks
    0
    Thanked 0 Times in 0 Posts
    On my computer the speed gap between the two versions is larger. Looks like the most improvement is gained on very redundant files (fp.log, rafael.bmp, a starting point for transforming the context scheme (the other thread) into a (smaller) state machine. In my yesterday experiments the performance impact was less critical when using a context of 3 to 4 bits, instead of 6.

    Something i forgot to mention is, that when you seperate the history maps, you can assign diffrent adaption speeds, since less predictable states should adapt slower to avoid the noise generated by nonstationarity (like compressing random data with a simple nonstationary model, which expands it).

    With a context of 3 bits, the speed was almost identical, since this cmm version is a slight rewrite of the previous one, grouping similar operations together and only using two function calls per coded character (the blank version, without the context sensitive history map ran at ~390kb/s vs 330kb/s with the old version).
    M1, CMM and other resources - http://sites.google.com/site/toffer86/ or toffer.tk

  7. #67
    Programmer toffer's Avatar
    Join Date
    May 2008
    Location
    Erfurt, Germany
    Posts
    587
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Here another quick release. I finally managed to get a proper SSE, my previous implementations just contained a tricky bug (written a "," instead of ";"). Speed and compression are improved. I think i almost squeezed out everything on the CM side, next is to make LZP work.

    http://freenet-homepage.de/toffer_86/cmm2-080113.7z

    N8
    M1, CMM and other resources - http://sites.google.com/site/toffer86/ or toffer.tk

  8. #68
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,257
    Thanks
    307
    Thanked 795 Times in 488 Posts

  9. #69
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Thanks Chris!

    Mirror: Download

  10. #70
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Quick Test...

    A10.jpg > 833,347
    AcroRd32.exe > 1,413,287
    english.dic > 550,229
    FlashMX.pdf > 3,686,716
    FP.LOG > 598,115
    MSO97.DLL > 1,773,988
    ohs.doc > 811,388
    rafale.bmp > 762,993
    vcfiu.hlp > 624,700
    world95.txt > 521,980

    Total = 11,576,743 bytes

  11. #71
    Programmer toffer's Avatar
    Join Date
    May 2008
    Location
    Erfurt, Germany
    Posts
    587
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Thanks a lot!
    M1, CMM and other resources - http://sites.google.com/site/toffer86/ or toffer.tk

  12. #72
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,912
    Thanks
    291
    Thanked 1,272 Times in 719 Posts

  13. #73
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Precomp v0.3.7 + CMM2-080113

    A10.jpg > 700,202 *
    AcroRd32.exe > 1,413,287
    english.dic > 550,229
    FlashMX.pdf > 2,589,078 *
    FP.LOG > 598,115
    MSO97.DLL > 1,773,988
    ohs.doc > 710,510 *
    rafale.bmp > 762,993
    vcfiu.hlp > 624,700
    world95.txt > 521,980

    Total = 10,245,082 bytes

  14. #74
    Programmer toffer's Avatar
    Join Date
    May 2008
    Location
    Erfurt, Germany
    Posts
    587
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Hello everyone!

    Here is a small cmm3 preview. It lacks most improvements i'm thinking of, at the moment (exspecially lzp). The main compression gain is caused by a simple match model (see my other post), a rewritten hash mapping and some minor changes.

    http://freenet-homepage.de/toffer_86...080129_test.7z
    M1, CMM and other resources - http://sites.google.com/site/toffer86/ or toffer.tk

  15. #75
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Thanks Chris!

  16. #76
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Quote Originally Posted by toffer
    Hello everyone!

    Here is a small cmm3 preview. It lacks most improvements im thinking of, at the moment (exspecially lzp). The main compression gain is caused by a simple match model (see my other post), a rewritten hash mapping and some minor changes.

    http://freenet-homepage.de/toffer_86...080129_test.7z
    Quick test...

    A10.jpg > 831,196
    AcroRd32.exe > 1,378,188
    english.dic > 512,159
    FlashMX.pdf > 3,673,536
    FP.LOG > 482,187
    MSO97.DLL > 1,751,757
    ohs.doc > 767,587
    rafale.bmp > 762,422
    vcfiu.hlp > 548,486
    world95.txt > 483,089

    Total = 11,190,607 bytes


    ENWIK8 > 22,235,659 bytes

    ENWIK9 >193,752,743 bytes

  17. #77
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    222
    Thanked 146 Times in 83 Posts
    Thanks Toffer! Nice !

  18. #78
    Programmer toffer's Avatar
    Join Date
    May 2008
    Location
    Erfurt, Germany
    Posts
    587
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Hi!

    Here's another testing release. It is compiled for around 400mb of RAM and adds some small tweaks (i tried the paq state machine instead of mine, which gave a slight compression gain normally, but hurts for less compressible files) - the main improvement is the optimized mmx code for mixing, which gives a speedup of around 15% on my computer. The small memory version is even faster - i could boost the speed from 320 kb/s to 400 kb/s (1.8GHz Sempron) - since it spends less time on cache misses (fewer RAM) and mixing is more significant.

    http://freenet-homepage.de/toffer_86...080207_test.7z

    I would be interested in a test on a large fileset, like Nania's MOC.

    Thanks in advance
    M1, CMM and other resources - http://sites.google.com/site/toffer86/ or toffer.tk

  19. #79
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    222
    Thanked 146 Times in 83 Posts
    Hi Toffer! Updated in MOC!

  20. #80
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    222
    Thanked 146 Times in 83 Posts
    SFC Test 11.123.559 bytes in 85 sec. (91 sec. decompression)

  21. #81
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Thanks Chris!

  22. #82
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,257
    Thanks
    307
    Thanked 795 Times in 488 Posts
    http://cs.fit.edu/~mmahoney/compression/text.html#1796

    ( Decompression time is probably high because I was watching a video while running cmm at low priority. I use timer 3.01 process time, but I guess it's not perfect )

  23. #83
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Quick test...

    A10.jpg > 834,990
    AcroRd32.exe > 1,368,984
    english.dic > 499,477
    FlashMX.pdf > 3,673,346
    FP.LOG > 468,700
    MSO97.DLL > 1,735,260
    ohs.doc > 766,537
    rafale.bmp > 757,139
    vcfiu.hlp > 540,661
    world95.txt > 478,079

    Total = 11,123,173 bytes

  24. #84
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Another quick test...

    Test machine: Intel PIII (Coppermine) @750 MHz, 512 MB RAM, Windows 2000 Pro SP4

    Test File: ENWIK8 (1,000,000,000 bytes)

    Timed with AcuTimer v1.2


    ENWIK8 > 21,212,766 bytes
    out/in: 20715/97656, 21.21% (1.70 pbc); speed: 120.1 kB/s
    Elapsed Time: 00:13:33.115 (813.115 Seconds)

  25. #85
    Programmer toffer's Avatar
    Join Date
    May 2008
    Location
    Erfurt, Germany
    Posts
    587
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Thanks!
    M1, CMM and other resources - http://sites.google.com/site/toffer86/ or toffer.tk

  26. #86
    Programmer toffer's Avatar
    Join Date
    May 2008
    Location
    Erfurt, Germany
    Posts
    587
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Good evening!

    I've completely changed my mixing architecture to NN's and added a simple mixing hierarchy. There's a match model, which uses a 16 mb sliding window. I use a simple heuristic to turn models off, when the higher order models seem to offer good predictions. The source has almost completely been rewritten. In the near future I'll add a compression option switch, which controls the memory allocated.

    Here's a testing release of cmm4:

    http://freenet-homepage.de/toffer_86/cmm4.exe

    And now there's a version number
    M1, CMM and other resources - http://sites.google.com/site/toffer86/ or toffer.tk

  27. #87
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Awesome! Thanks toffer!

  28. #88
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Quick test...

    A10.jpg > 831,088
    AcroRd32.exe > 1,337,313
    english.dic > 503,406
    FlashMX.pdf > 3,659,660
    FP.LOG > 485,520
    MSO97.DLL > 1,701,679
    ohs.doc > 759,871
    rafale.bmp > 746,667
    vcfiu.hlp > 530,022
    world95.txt > 476,505

    Total = 11,031,731 bytes

  29. #89
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Another quick test...

    Test machine: Intel PIII (Coppermine) @750 MHz, 512 MB RAM, Windows 2000 Pro SP4

    Test File: ENWIK8 (1,000,000,000 bytes)

    Timed with AcuTimer v1.2


    ENWIK8 > 21,459,665 bytes
    Elapsed Time: 00:10:56.384 (656.384 Seconds)

  30. #90
    Programmer toffer's Avatar
    Join Date
    May 2008
    Location
    Erfurt, Germany
    Posts
    587
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Thanks!

    cmm3 080207 - 813.115 Seconds
    cmm4 v0.0 - 656.384 Seconds

    -> Around 20% faster without any ASM

    BTW: I tested this against a ripped off version of lpaq (static counters, no second SSE stage and text model - which equals my layout). Results are almost equal, only 10k difference, while beeing almost twice as fast (at least on my machine).
    M1, CMM and other resources - http://sites.google.com/site/toffer86/ or toffer.tk

Page 3 of 6 FirstFirst 12345 ... LastLast

Similar Threads

  1. Simple bytewise context mixing demo
    By Shelwien in forum Data Compression
    Replies: 13
    Last Post: 3rd April 2020, 16:04
  2. another (too) fast compressor
    By Cyan in forum Data Compression
    Replies: 139
    Last Post: 6th February 2016, 20:41
  3. Context mixing
    By Cyan in forum Data Compression
    Replies: 7
    Last Post: 4th December 2009, 18:12
  4. PACKET v.0.01 new fast compressor !
    By Nania Francesco in forum Data Compression
    Replies: 45
    Last Post: 19th June 2008, 01:44
  5. Fast PPMII+VC Compressor
    By in forum Forum Archive
    Replies: 4
    Last Post: 2nd August 2006, 19:17

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •