Results 1 to 9 of 9

Thread: LZP flag sequence compression

  1. #1
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,366
    Thanks
    212
    Thanked 1,018 Times in 540 Posts

    LZP flag sequence compression

    "pinc" on IRC posted a sample flag sequence from his LZP implementation.
    And surprisingly paq8 shows much worse results than 7-zip for it,
    so I got interested and experimented a little:

    http://ctxmodel.net/files/mix_test/m...t_vA2-pinc.rar
    Code:
    Experiment with CM tuning for LZP control sequence
    
    779280 pinc_sample 
     68753 paq8p -6
     62166 7z 9.04 -mx=9
     60924 7z ??? (reported by chornobyl)
     71329 mix_test 09h2x62
     59988 mix_test 09h2x6a
    
    09h2x62 - single counter / static allocation
    09h2x6a - 8 counters / dynamic allocation
    (59988 is an intermediate result, current one is already 59930,
    but I didn't bother to update it).

    The mix_test result shows that its probably not a matter
    of mixing vs LZ optimal parsing, as mixing still wins.
    But then what's the matter with paq8?
    Last edited by Shelwien; 22nd July 2009 at 23:03.

  2. #2
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 779 Times in 486 Posts
    Maybe paq8 lacks an order-17 model.
    Code:
    C:\tmp\mix_test_vA2-pinc>ppmonstr e -o16 -fpinc_sample.pmm-o16 pinc_sample
    Monstrous PPMII compressor based on PPMd var.J, Feb 16 2006
       pinc_sample: 779280 >  76587, 0.786 bpb, used:  2.5MB, speed: 686 KB/sec
    
    C:\tmp\mix_test_vA2-pinc>ppmonstr e -o17 -fpinc_sample.pmm-o17 pinc_sample
    Monstrous PPMII compressor based on PPMd var.J, Feb 16 2006
       pinc_sample: 779280 >  63122, 0.648 bpb, used: 24.4MB, speed: 551 KB/sec
    I also looked at the file with fv, but the order-17 structure was not obvious.

  3. #3
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    956
    Thanks
    569
    Thanked 395 Times in 293 Posts
    Hi!
    Iheve question - don't blame me if it's stupid: are you sure that output file from 09h2x6a is correct? Have you compared uncompressed files with original (CRC)?

    I've tried to use this version on my testbed files and all scores are better than paqxxx compressed files and ... it's strange.

    It looks like some part of files are missed in transformation or I'm wrong and this version of r1c1 file couldn't be used to other files than original book1bwt or pinc_sample...

    Darek

  4. #4
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,366
    Thanks
    212
    Thanked 1,018 Times in 540 Posts
    Well, obviously its a flag model... it encodes only bit0 of each byte.
    Nobody said that it can be used with other files and/or would be always better than paq8

  5. #5
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    956
    Thanks
    569
    Thanked 395 Times in 293 Posts
    Ok,
    sorry for question.

  6. #6
    Member chornobyl's Avatar
    Join Date
    May 2008
    Location
    ua/kiev
    Posts
    153
    Thanks
    0
    Thanked 0 Times in 0 Posts
    best result so far
    Attached Files Attached Files

  7. #7
    Member Skymmer's Avatar
    Join Date
    Mar 2009
    Location
    Russia
    Posts
    681
    Thanks
    38
    Thanked 168 Times in 84 Posts
    Quote Originally Posted by Shelwien View Post
    "pinc" on IRC posted a sample flag sequence from his LZP implementation.
    And surprisingly paq8 shows much worse results than 7-zip for it,
    so I got interested and experimented a little...
    Realy interesting sample. I did a couple of tests with some other programms.
    -6 level used for PAQs. 512MB\Fast Analysis for PWCM and 256MB\512 (Largest optimised match)\Fast Analysis for ROLZ.
    Code:
    paq8p           68 753
    paq8p3          68 665
    paq8px          69 013
    paq8k           78 133
    winrk_pwcm      73 555
    winrk_rolz3     59 131
    Quote Originally Posted by chornobyl
    best result so far...
    Damn! 60 468 bytes with 7z its realy good! I'm still trying to beat this result but probably you have used some tricks. At least one surely. I mean that you somehow managed to make the file inside archive to have no name
    Anyway, my best result so far is 60 476 bytes.

  8. #8
    Member chornobyl's Avatar
    Join Date
    May 2008
    Location
    ua/kiev
    Posts
    153
    Thanks
    0
    Thanked 0 Times in 0 Posts
    If your result is 60476 than you probably win, because my version with name takes 60487, and to remove name i used 7zFM. Also xz variant is even smaller because of smaller headers.

  9. #9
    Member Skymmer's Avatar
    Join Date
    Mar 2009
    Location
    Russia
    Posts
    681
    Thanks
    38
    Thanked 168 Times in 84 Posts
    Quote Originally Posted by chornobyl View Post
    If your result is 60476 than you probably win, because my version with name takes 60487, and to remove name i used 7zFM. Also xz variant is even smaller because of smaller headers.
    First of all thanks for "rename" tip! It helped a little but I was able to reach 60 469 only which is 1 byte larger than yours. Damn! 1 byte again
    Second, the probably term is not an option for me.
    Third, after some additional bruteforcing I was able to produce 60 416 bytes version. By the way not with v9.04 and not even with v4.65
    Attached Files Attached Files

Similar Threads

  1. Sequence of bits
    By Kaw in forum Data Compression
    Replies: 12
    Last Post: 25th September 2009, 09:53
  2. flzp, new LZP compressor/preprocessor
    By Matt Mahoney in forum Data Compression
    Replies: 13
    Last Post: 23rd June 2008, 18:24
  3. FLASHZIP new ARI+LZP compressor
    By Nania Francesco in forum Forum Archive
    Replies: 65
    Last Post: 5th February 2008, 22:42

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •