Results 1 to 14 of 14

Thread: lzpm v0.16 is here!

  1. #1
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,979
    Thanks
    376
    Thanked 347 Times in 137 Posts

    Talking lzpm v0.16 is here!

    Please welcome the strongest LZ77 of my own.

    What's new/features:
    • LZ77 with 8 MB dictionary
    • Enhanced S&S parsing on 32 MB blocks - i.e. at each buffer position we find not just one but a few good matches for further path optimization
    • New match finder: HC5 - Hash Chains with 3-4-5-byte hashing
    • Enhanced LZ-output coding
    • Two compression modes: Normal ("e") and Extreme ("ex") for compression maniacs

    Attached Files Attached Files

  2. #2
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by encode View Post
    Please welcome the strongest LZ77 of my own.

    What's new/features:
    • LZ77 with 8 MB dictionary
    • Enhanced S&S parsing on 32 MB blocks - i.e. at each buffer position we find not just one but a few good matches for further path optimization
    • New match finder: HC5 - Hash Chains with 3-4-5-byte hashing
    • Enhanced LZ-output coding
    • Two compression modes: Normal ("e") and Extreme ("ex") for compression maniacs

    Thanks.
    Quick test on Shelwein's Bookstar:
    11358322B.

    For comparison:
    FreeArc LZMA:max:BT4 10746537
    FreeArc LZMA:max:HT4 11669215
    rzm: 10313547
    LZTurbo -59: 10958180

  3. #3
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts

    Thumbs up

    Quote Originally Posted by encode View Post
    Please welcome the strongest LZ77 of my own.

    What's new/features:
    • LZ77 with 8 MB dictionary
    • Enhanced S&S parsing on 32 MB blocks - i.e. at each buffer position we find not just one but a few good matches for further path optimization
    • New match finder: HC5 - Hash Chains with 3-4-5-byte hashing
    • Enhanced LZ-output coding
    • Two compression modes: Normal ("e") and Extreme ("ex") for compression maniacs

    Thanks Ilia!

    Mirror: Download

  4. #4
    Programmer osmanturan's Avatar
    Join Date
    May 2008
    Location
    Mersin, Turkiye
    Posts
    651
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Tested on Core 2 Quad 2.4 GHz, 4 GB RAM, Vista Ultimate x64
    Code:
                  LZPM 0.16 (-e)   LZPM 0.16 (-ex)   WinRAR 3.70 (Best)
    -------------------------------------------------------------------
    a10.jpg             836,552           836,552               842,534
    AcroRd32.exe      1,395,428         1,394,599             1,376,734
    english.dic         885,771           885,759             1,114,867
    FlashMX.pdf       3,748,589         3,747,952             3,749,331
    FP.pdf              828,820           785,296               665,170
    MSO97.dll         1,828,045         1,827,441             1,853,287
    ohs.doc             815,136           811,218               806,708
    rafale.bmp        1,023,495         1,022,087               821,991
    vcfiu.hlp           676,282           674,223               668,617
    world95.txt         588,652           583,478               464,069
    Total Size       12,626,770        12,568,605            12,363,308
    Total Time      124.632 sec       (VERY LONG)            16.090 sec
    I found even (-e) mode slow enough. But, (-ex) mode has nearly PAQ8 like speed performance I know, you left this option for "compression maniacs". But, the compression score is not very different from (-e) mode while being hundred times slower. This is not good for a LZ coder (As a note, CMM4 is twice faster than even -e mode while having ~2 MB better compression). I think, you should remove (-ex) mode and make a new fast mode (faster than -e mode). I didn't test 7-zip because it would be unfair.

  5. #5
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,979
    Thanks
    376
    Thanked 347 Times in 137 Posts
    I know that the compression difference between "e" and "ex" is small, I kept "ex" as an uncompromised variant - to get the deepest search from hash chains. Anyway, even multi-level hash chains are not intended for optimal parsing and such large dictionaries.

  6. #6
    Member chornobyl's Avatar
    Join Date
    May 2008
    Location
    ua/kiev
    Posts
    153
    Thanks
    0
    Thanked 0 Times in 0 Posts
    tried to test on my usual 512K binary file and failed, lzpm apeeared to be too memory hungry for my 512M of ram(384free) even in e mode. It wont run until i increase swap file to 512M and then with hdd thrashing tested on smaller file
    64.raw 256x256 8bit raw image
    Code:
    unk        65536
    lzmp016 e  51409 122s
    lzmp016 ex 51409 118s
    fpaq0f2    50999   1s
    fpaq02     62373   2s
    Havent you think about some f mode for faster compression and lower memory usage, while keeping e and ex modes as well.

  7. #7
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,979
    Thanks
    376
    Thanked 347 Times in 137 Posts

    Wink

    Yep, you need at least 1 GB RAM, which should be OK these days.

    Don't forget - the catch is fast enough DEcompression.

    "e" and "ex" modes use the same amount of memory with the same path optimization the only difference in hash chain length:

    e - 256
    ex - 16K



    Well, this is an experimental release...

  8. #8
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    735
    Thanked 660 Times in 354 Posts
    for me, ex-like modes are good for testing strength of model, with the perfect match finder. so 1) you know how much you lose due to imperfectness of real matchfinder, 2) you may compare different coders ignoring their MF differences. so, it just targets other goal

  9. #9
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,979
    Thanks
    376
    Thanked 347 Times in 137 Posts

    Cool

    Just started working on the lzpm v0.17. New lzpm should be FAST! It is a complete rewrite of the compressor and it will have all new code parts and features:
    • The FAST, newest and the most advanced LZP-based layer (32 MB dictionary)
    • Extremely fast CM back end
    • The standard E8E9 transformer
    This is my first LZ-based compressor ever that will be fully optimized with my all new and special program optimizer. The BCM power mostly came from that optimizer!
    The fastest LZ-layer came from my new and non-public fast compressor (the competitor to the THOR and others). I tried ALL possible LZ-driven ideas - LZ77, LZRW, ROLZ, LZP, ... done some theoretical conclusions about the algorithms, what leads to the best results in each case, what is the best feature of each modification, etc. And came up with this brilliant compression piece. After all, I decided to add some arithmetic, plus an optimized and simple enough modeler to get even more breathtaking results!
    Additionally, I just realized that PAQ-speed compressors are out of business! We need fast, simple and yet efficient compression. By combining all my experience from LZ and CM areas (actual CM part is even further improved CM from the BCM!) we may get something very special!
    (Drinking Vodka+Cola always helps!)


  10. #10
    Programmer osmanturan's Avatar
    Join Date
    May 2008
    Location
    Mersin, Turkiye
    Posts
    651
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Do not drink to much else you can accidentally delete new LZPM's sources Anyway, I look forward to see new LZPM. If everything will be as you mentioned, my work-in-progress LZ codecs (a fast and slow variant) will have a new "competitor"
    BIT Archiver homepage: www.osmanturan.com

  11. #11
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Quote Originally Posted by osmanturan View Post
    my work-in-progress LZ codecs (a fast and slow variant) will have a new "competitor"
    That should be interesting!

  12. #12
    Member Fu Siyuan's Avatar
    Join Date
    Apr 2009
    Location
    Mountain View, CA, US
    Posts
    176
    Thanks
    10
    Thanked 17 Times in 2 Posts

    Smile

    LZP?does it mean slower decompression speed than your other LZ coders? i am waiting for your new program! i also want to be your competitor.....but so busy in GRE and final exam these days .....

  13. #13
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,979
    Thanks
    376
    Thanked 347 Times in 137 Posts

    Cool

    Quote Originally Posted by osmanturan View Post
    Do not drink to much else you can accidentally delete new LZPM's sources
    It doesn't matter - the entire source is in my head! That fact is the most important!

    Quote Originally Posted by osmanturan View Post
    Anyway, I look forward to see new LZPM. If everything will be as you mentioned, my work-in-progress LZ codecs (a fast and slow variant) will have a new "competitor"
    New LZPM has no competition! I'm just not really sure about that here are too many people that are experienced enough and have such programming skills to catch the uncatchable Joe!
    Quote Originally Posted by ForTheKing
    LZP?does it mean slower decompression speed than your other LZ coders?
    Not really, the decompression speed is fast enough - just slightly slower than the latest and final BALZ - it's due to the more heavier CM backend. As I said, the catch is fast compression, not only the decompression.

    Anyway, I'm continue the design - many stuff might be changed easily...


    Ah, forgot about my Vodka+Pepsi

  14. #14
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,979
    Thanks
    376
    Thanked 347 Times in 137 Posts
    a group of cowboys sitting in the bar. Suddenly someone dashes into the bar, runs around and dashes out.
    New cowboy asks: "who is that?"
    - "this is Uncatchable Joe, nobody ever managed to catch him!"
    - "why did noone catch him?"
    - "who would bother doing that?.."


Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •