Page 2 of 3 FirstFirst 123 LastLast
Results 31 to 60 of 64

Thread: PAQ8K

  1. #31
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,217
    Thanks
    743
    Thanked 495 Times in 383 Posts
    Hi! Thanks LovePimple for these speedup version!

    I've tested first speed optimised Paq8k2 versions - lp and lp2. Both are about 9% faster than original paq8k2.

    Maybe it doesn't look impressive for someone, but any % of paq8k2 speedup is worth mentioning. To test original paq8k2 version on my testbed set of files I've wasted about 100 hours.... and it could be 9 hours less with this version.

    Darek

  2. #32
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Thanks for the feedback! Are my latest experimental builds (download from SendSpace) any faster?

  3. #33
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,217
    Thanks
    743
    Thanked 495 Times in 383 Posts
    I'm still testing.
    For raffly lookup: paq8k2_opt4 version shows the best scores - about 1.5% faster than paq8k2_lp. It means 10-11% speedup from original paq8k2.
    Darek

  4. #34
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,217
    Thanks
    743
    Thanked 495 Times in 383 Posts
    Hi!
    some feedback about new compilations. Due to very slow algorithm I've tested only some files.
    In average the best compilation is opt3 version which speedup compression about 15%! However some files (textual) could speedup up to about 21%! Exe file have got minor speedup - 12.5%.

    Opt3 version wins becouse shows the biggest speedup on the biggest files, but opt4 is faster on smaller files.

    In attached xls are detailed scores for 7 my testbed files.

    Darek
    Attached Files Attached Files

  5. #35
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts

    Thumbs up

    Thanks for testing!

  6. #36
    Member Skymmer's Avatar
    Join Date
    Mar 2009
    Location
    Russia
    Posts
    688
    Thanks
    41
    Thanked 173 Times in 88 Posts
    A couple of modest speed tests for PAQ8k2 -6 on AMD Athlon 64 4000+ (Single core)
    Code:
                 |    #1   |    #2   |
    -------------|---------|---------|
    paq8k2_nodll | 623.151 | 570.387 |
    paq8k2_lp    | 582.934 | 537.639 |
    paq8k2_lp2   | 590.837 | 543.733 |
    paq8k2_opt1  | 583.208 | 538.038 |
    paq8k2_opt2  | 590.320 | 542.906 |
    paq8k2_opt3  | 591.061 | 543.677 |
    paq8k2_opt4  | 590.083 | 543.285 |
    ---------------------------------'
    6.45% speedup for #1 and 5.74% for #2. Average speedup is 6.1%.
    LP, once again, my compliments

  7. #37
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Bill Pettis has released PAQ8K3...

    http://ilovemyking.googlepages.com/paq8k3.cpp
    http://ilovemyking.googlepages.com/paq8k3.exe


    Thanks to Matt Mahoney for the links.

  8. #38
    Member
    Join Date
    Aug 2009
    Location
    Bari
    Posts
    74
    Thanks
    1
    Thanked 1 Time in 1 Post
    wooow!! it's fast than old versions!! The compression ratio is abnormal!! I have never seen anything else! thanks Matt Mahoney and Bill Pettis.

  9. #39
    Programmer Jan Ondrus's Avatar
    Join Date
    Sep 2008
    Location
    Rychnov nad Kněžnou, Czech Republic
    Posts
    279
    Thanks
    33
    Thanked 138 Times in 50 Posts

    paq8kx_v4 and paq8kx_v5

    Description:
    paq8kx_v4 = paq8kx_v1 + paq8px_v68
    paq8kx_v5 = paq8kx_v1 + paq8px_v68p3
    Attached Files Attached Files

  10. #40
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts

    Thumbs up

    Thanks Jan!

    Here are my usual builds...

    NOTE: _speed_optimised is now renamed _alt (alternative).
    Attached Files Attached Files

  11. #41
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,217
    Thanks
    743
    Thanked 495 Times in 383 Posts
    Thanks Jan for new, very powerful versions.
    Thanks LovePimple for comilations.

    My tests are in attached excel file. To short sum up:

    paq8kx_v4 -GREAT version, which breaks almost all my records for files which haven't any special/dedicated compression type in paq algorithm. However - there is one backwardness to old paq8kx_v1 version - all dedicated/specified compressions methods (WAV, GRAPHIC (BMP, TIFF, TGA), JPG, DBF) are (it looks like for me) non-changed to paq8kx_v1. Due to this, compression ratio of files which use this special compression methods is worse than in standard V68 version. Special compression methods for my testbed are statistically the best in mentioned standard V68 version.

    paq8kx_v5 - it is noticeably weaker version than v4. In overall is better than V68, however for files which didn't used special compressions, score is worse than paq8px_v1.

    Darek
    Attached Files Attached Files

  12. #42
    Member
    Join Date
    Sep 2009
    Location
    USA
    Posts
    16
    Thanks
    0
    Thanked 0 Times in 0 Posts
    I was happy to find this, here's my test:

    Notepad2.exe (x64 version)
    ----------------------

    paq8k3 -6: 257k (100%)
    pxv68p3 6: 235k (91.4%)
    paq8k5 -6: 230k (89.5%)
    paq8k4 -6: 229k (89.1%)

    Nice work.

  13. #43
    Member
    Join Date
    Sep 2009
    Location
    USA
    Posts
    16
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Mixer update hack, saves a few hundred bytes here.
    Attached Files Attached Files

  14. #44
    Member
    Join Date
    Sep 2009
    Location
    USA
    Posts
    16
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Well, now that "err" is random, I realized that using 2 mixers to do the exact same thing will produce different results. So here it is with 2 identical mixers. It's very slow now, and compresses very slightly better.
    Attached Files Attached Files

  15. #45
    Member
    Join Date
    May 2009
    Location
    China
    Posts
    36
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Bill Pettis View Post
    Well, now that "err" is random, I realized that using 2 mixers to do the exact same thing will produce different results. So here it is with 2 identical mixers. It's very slow now, and compresses very slightly better.
    Thanks

  16. #46
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,217
    Thanks
    743
    Thanked 495 Times in 383 Posts
    Hi!
    There are results of my test of 4a and 4adouble2 versions (in excel file).

    To short sum up paq8kx_v4a is an inprovement to version 4 and paq8kx_v4adouble2 shows even more improvement to version 4a. Another record breaking versions! Double2 version is about 2 times slower than version 4, but for bigger files become very, very slow - 3-4 times slower than 4.

    Files w/o special compression methods are compressed better than version 4 and even more in comparison vs. v68.
    For files with dedicated compression methods (WAV, GRAPHIC (BMP, TIFF, TGA), JPG, DBF) compression is worse than v68 in all cases, and very often worse than previous v4 version.

    Darek
    Attached Files Attached Files

  17. #47
    Member
    Join Date
    Sep 2009
    Location
    USA
    Posts
    16
    Thanks
    0
    Thanked 0 Times in 0 Posts
    I also tried doubling the number of "m.set" commands via a loop, but the results were not as well. Apparently there is a need for 2 mixers, as my previous effort. It continues to improve with four mixers, and so on, but I wouldn't release a program with that many mixers...except as a joke.

  18. #48
    Member
    Join Date
    Sep 2009
    Location
    USA
    Posts
    16
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Thumbs up

    No code, more rambling:

    Using 2 identical mixers works great, and differentiating the mixers usually makes things worse. The thing that can change between them is the pseudo-context "order" in ContextModel2. There are many possibilities for how to calculate this--usually by taking the return value from from another model instead of the Normal Model.

  19. #49
    Member
    Join Date
    Sep 2009
    Location
    USA
    Posts
    16
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Bill Pettis View Post
    No code, more rambling:

    Using 2 identical mixers works great, and differentiating the mixers usually makes things worse. The thing that can change between them is the pseudo-context "order" in ContextModel2. There are many possibilities for how to calculate this--usually by taking the return value from from another model instead of the Normal Model.
    Here's an example of what I just explained.

    I think there is a possibility for further differentiation of the mixers, but I am frustrated and going to leave it alone for now. Examine the process used for the determination of order, and change it if you can. I am still assuming that if the two mixers were not identical the compression may improve.
    Attached Files Attached Files

  20. #50
    Programmer Jan Ondrus's Avatar
    Join Date
    Sep 2008
    Location
    Rychnov nad Kněžnou, Czech Republic
    Posts
    279
    Thanks
    33
    Thanked 138 Times in 50 Posts
    Quote Originally Posted by Bill Pettis View Post
    I think there is a possibility for further differentiation of the mixers, but I am frustrated and going to leave it alone for now. Examine the process used for the determination of order, and change it if you can. I am still assuming that if the two mixers were not identical the compression may improve.
    Code:
    int pr=m2[0].p()+m2[1].p()+1>>1;
    What about some more advanced combination of Mixers outputs than simple average?

  21. #51
    Member
    Join Date
    Sep 2009
    Location
    USA
    Posts
    16
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Jan Ondrus View Post
    What about some more advanced combination of Mixers outputs than simple average?
    It's possible to make another mixer to combine the outputs of several mixers, I suppose. I tried this a long time ago, but no luck recently.

  22. #52
    Member
    Join Date
    Sep 2009
    Location
    USA
    Posts
    16
    Thanks
    0
    Thanked 0 Times in 0 Posts
    I think this should have better compression than the rest of the kx versions in this thread. It is also a nice speed-up compared to using 2 mixers.
    Attached Files Attached Files

  23. #53
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,217
    Thanks
    743
    Thanked 495 Times in 383 Posts
    Hi!
    my scores for Paq8kx_v6 are in enclosed file.

    For whlole testbed compression ratio is almost identical as for Paq8kx_v4double2 (108 bytes more). Some files are compressed slightly better, some worse than Paq8kx_v4double2, but differences arent higher than some hundreads of bytes - even for bigger files. Compression times are higher than double2 version.

    One thing worth to mention - all text files are compressed slightly better than other kx and px versions.

    Darek
    Attached Files Attached Files

  24. #54
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,217
    Thanks
    743
    Thanked 495 Times in 383 Posts
    And paq8kx_v7 scores - in attached file.

    Version 7 has some nice improvements to v6 or 4double2, but only for default compression (not defined files), which cover text files too. Yet another record version!

    However, due to some reasons each next version of kx have poorer compression of special files (wave and graphic) than previous. Better default compression causes in worse special compression scores. Due to this effect, for whole test gain from default compression is almost compensated by special files loss and difference between previous and verion 7 for total score is very small.

    Darek
    Attached Files Attached Files

  25. #55
    Member
    Join Date
    Sep 2009
    Location
    USA
    Posts
    16
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Darek View Post
    Compression times are higher than double2 version.
    That surprised me. But I only ever test very small files. You can't expect me to write code AND test it. :P

  26. #56
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,217
    Thanks
    743
    Thanked 495 Times in 383 Posts
    To be honest - I'm only compression ratio freak because compression ratio is the only variable wich is constant over time. I've mentioned compression times approximation to raffly comparison - slower/faster.

    Compression time isn't any important factor for me because his scores are only contemporary proper. Of course there are some limitations on extremally slow programs, but in longer time perspective this variable is negligible.

    Darek

  27. #57
    Member
    Join Date
    Sep 2009
    Location
    USA
    Posts
    16
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Darek View Post
    To be honest - I'm only compression ratio freak because compression ratio is the only variable wich is constant over time. I've mentioned compression times approximation to raffly comparison - slower/faster.

    Compression time isn't any important factor for me because his scores are only contemporary proper. Of course there are some limitations on extremally slow programs, but in longer time perspective this variable is negligible.

    Darek
    I agree, mostly. I think that as the algorithm improves, the same compression will be attained at faster speeds. I remember PAQ7, when the match model completely took over on a match, causing a huge speed-up but hurting compression later, as the mixer wasn't trained on all those skipped predictions. Somehow there has to be a way to achieve speed and high compression. I'm certainly not looking for it in paq8k, though. Not yet.

  28. #58
    Member
    Join Date
    Dec 2009
    Location
    Canada
    Posts
    6
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Hello, I'm quite new to coding, but I do know some of the basics. I have a problem with compiling, and the compiler returns errors "undefined reference to 'dot_product'" and "undefined reference to 'train'". Am I missing a file?


    Thanks,
    SFI

  29. #59
    Member
    Join Date
    May 2008
    Location
    England
    Posts
    325
    Thanks
    18
    Thanked 6 Times in 5 Posts
    dot_product, i think that's in the asm file? been a while since i compiled PAQ.
    You should be able to compile PAQ without it though, details are in the cpp file as to how.

    The filename is paq7asm_sse2.asm and you can find it inside the zip here:
    http://encode.dreamhosters.com/showp...&postcount=573

  30. #60
    Member
    Join Date
    Dec 2009
    Location
    Canada
    Posts
    6
    Thanks
    0
    Thanked 0 Times in 0 Posts
    I tried the file paq7asm_sse.asm and this is the error I have:
    Code:
    /usr/bin/ld:paq7asm_sse2.asm: file format not recognized; treating as linker script
    Then I tried paq7asm.obj, and this is the error I have:

    Code:
    paq7asm.obj: file not recognized: File format not recognized
    I'm not sure what happened.

Page 2 of 3 FirstFirst 123 LastLast

Similar Threads

  1. PAQ8K
    By Black_Fox1 in forum Forum Archive
    Replies: 10
    Last Post: 15th February 2007, 22:57

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •