Page 68 of 71 FirstFirst ... 18586667686970 ... LastLast
Results 2,011 to 2,040 of 2112

Thread: paq8px

  1. #2011
    Member Gotty's Avatar
    Join Date
    Oct 2017
    Location
    Switzerland
    Posts
    552
    Thanks
    356
    Thanked 355 Times in 192 Posts
    Luca, what is the big red part in the second image (v189 -9)?

  2. #2012
    Member
    Join Date
    Sep 2014
    Location
    Italy
    Posts
    94
    Thanks
    102
    Thanked 39 Times in 25 Posts
    Sorry Gotty,
    It is not very clear..
    Values rapresents the difference with
    188b -9l (in the first image attached)
    Luca

  3. #2013
    Member Gotty's Avatar
    Join Date
    Oct 2017
    Location
    Switzerland
    Posts
    552
    Thanks
    356
    Thanked 355 Times in 192 Posts
    I had a feeling like that. But comparing 9L -> 9 is not fair. From the red numbers it looks like that there is a serious degradation between the two versions which is not the case.
    Comparing 188 -9 -> 189 -9 is fair and comparing
    188 -9L -> 189 -9L is fair. And comparing any x->xL is also useful to see how much LSTM helps.
    Could you fix that in some way?
    And a "cosmetics" wish: could you add headers to the tables so we know which column is which?

  4. #2014
    Member
    Join Date
    Sep 2014
    Location
    Italy
    Posts
    94
    Thanks
    102
    Thanked 39 Times in 25 Posts
    Ok perfect! I will do it!
    Thank you!

  5. #2015
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,217
    Thanks
    743
    Thanked 495 Times in 383 Posts
    These are scores of 4 corpuses by v188, v188 LSTM, v188b LSTM and v189 LSTM.
    For Silesia corpus there are gains for every new version with LSTM.
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	paq8px_v189_4_Corpuses.jpg 
Views:	45 
Size:	2.66 MB 
ID:	7828  

  6. Thanks:

    LucaBiondi (30th July 2020)

  7. #2016
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,217
    Thanks
    743
    Thanked 495 Times in 383 Posts
    enwik scores for the latest versions of paq8px:

    16'190'519 - enwik8 -12 by Paq8px_v187fix2, change: 0,00% - this version changes are compared to paq8px_v183fix1
    16'080'588 - enwik8 -12eta by Paq8px_v187fix2, change: -1,74%
    15'889'931 - enwik8.drt -12eta by Paq8px_v187fix2, change: -1,18%
    127'626'051 - enwik9_1423 -12eta by Paq8px_v187fix2, change: -4,37%
    124'786'260 - enwik9_1423.drt -12eta by Paq8px_v187fix2, change: -4,07%


    15'900'206 - enwik8 -12leta by Paq8px_v188, change: -1,12%
    15'503'221 - enwik8.drt -12leta by Paq8px_v188, change: -2,43%

    15'907'081 - enwik8 -12leta by Paq8px_v188b, change: 0,04%
    15'505'761 - enwik8.drt -12leta by Paq8px_v188b, change: 0,02%

    15'896'588 - enwik8 -12leta by Paq8px_v189, change: -0,07%
    15'490'302 - enwik8.drt -12leta by Paq8px_v189, change: -0,10%

    My raffle estimate of pure enwik9 is about 126'2xx'xxx bytes and DRT version about 121'6xx'xxx bytes.

  8. Thanks:

    LucaBiondi (30th July 2020)

  9. #2017
    Member
    Join Date
    Aug 2017
    Location
    United States
    Posts
    33
    Thanks
    19
    Thanked 23 Times in 16 Posts
    @Darek is it possible for me to download your testset? I would like to do some testing of my own, and I've searched the forum but haven't found it available.

    @Gotty/@mpais: Glad to see all the recent activity from you two

  10. #2018
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,217
    Thanks
    743
    Thanked 495 Times in 383 Posts
    Quote Originally Posted by Eppie View Post
    @Darek is it possible for me to download your testset? I would like to do some testing of my own, and I've searched the forum but haven't found it available.
    Yes, there was available partially. It's not official, there no internet place to get it.

    Here you are the testbed. The files and the description I've made.

    Due to fact that some files was get from existed programs from 1996 or images from non-verified sources then all files are for testing purpose only.
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	DBA_testset_description.jpg 
Views:	29 
Size:	277.2 KB 
ID:	7830  
    Attached Files Attached Files
    Last edited by Darek; 31st July 2020 at 21:38.

  11. Thanks (4):

    CompressMaster (31st July 2020),Eppie (31st July 2020),LucaBiondi (30th July 2020),schnaader (30th July 2020)

  12. #2019
    Member
    Join Date
    Jun 2009
    Location
    Puerto Rico
    Posts
    251
    Thanks
    138
    Thanked 52 Times in 39 Posts
    It's been over 3 days now compressing with v189 using LSTM and the compression progress is at 17%. Since v188 took 3 days, I wonder if this could be a result of:

    1. using an AMD CPU
    2. the new 24-bit images tweak.

    I'll probably abort some tasks and run on an intel CPU and see how long it takes there.

    I'm compressing the same files I compressed on v188 with LSTM.

  13. #2020
    Member
    Join Date
    Sep 2014
    Location
    Italy
    Posts
    94
    Thanks
    102
    Thanked 39 Times in 25 Posts
    maybe are you running out of memory?

  14. #2021
    Member
    Join Date
    Jun 2009
    Location
    Puerto Rico
    Posts
    251
    Thanks
    138
    Thanked 52 Times in 39 Posts
    Nope. I have 128GB RAM with over 60GB free RAM

  15. #2022
    Member CompressMaster's Avatar
    Join Date
    Jun 2018
    Location
    Lovinobana, Slovakia
    Posts
    198
    Thanks
    58
    Thanked 15 Times in 15 Posts
    Quote Originally Posted by Darek View Post
    Yes, there was available partially. It's not official, there no internet place to get it. Here you are the testbed. The files and the description I've made.
    @Darek, you could slightly decrease upload filesize if you compress DBA corpus as RAR with highest settings via winrar. I was able to shrink it down to 17,694,628 bytes.
    Please hit the "THANKS" button under my post if its useful for you.

  16. Thanks:

    Darek (31st July 2020)

  17. #2023
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,217
    Thanks
    743
    Thanked 495 Times in 383 Posts
    @ClmpressMaster - I'll do that. I'll also check latest 7zip option. Done. Scores for different archivers:

    40'405'504 - tar version
    20'762'347 - zip version - my previous file
    18'547'630 - 7zip bzip2 version
    17'951'250 - 7zip ppm2 version
    17'159'665 - Winrar - rar version
    16'103'044 - 7zip - lzma version
    16'083'346 - 7zip - lzma2 version
    15'625'846 - Winrar - rar4 version - uploaded

    but: score for Winrar 3.70b5 (max options for particular files) = 14'956'926

    @moisesmcardona - how much slower is the -l option? It should be about 3 - 3,5 times slower than non LSTM.
    Last edited by Darek; 31st July 2020 at 21:39.

  18. #2024
    Member Gotty's Avatar
    Join Date
    Oct 2017
    Location
    Switzerland
    Posts
    552
    Thanks
    356
    Thanked 355 Times in 192 Posts
    Quote Originally Posted by moisesmcardona View Post
    It's been over 3 days now compressing with v189 using LSTM and the compression progress is at 17%. Since v188 took 3 days, I wonder if this could be a result of:

    1. using an AMD CPU
    2. the new 24-bit images tweak.

    I'll probably abort some tasks and run on an intel CPU and see how long it takes there.

    I'm compressing the same files I compressed on v188 with LSTM.
    Sounds strange, indeed.
    I believe, using an AMD is not a problem. Does it have AVX2? If you use the -v option when
    start compressing your files and
    does it report that it uses AVX2?
    You may benefit from recompiling the source with -march=native and -mtune=native.
    Remark: the tweaks in v189 do not affect compression speed.

  19. #2025
    Member
    Join Date
    Jun 2009
    Location
    Puerto Rico
    Posts
    251
    Thanks
    138
    Thanked 52 Times in 39 Posts
    Quote Originally Posted by Gotty View Post
    Sounds strange, indeed.
    I believe, using an AMD is not a problem. Does it have AVX2? If you use the -v option when
    start compressing your files and
    does it report that it uses AVX2?
    You may benefit from recompiling the source with -march=native and -mtune=native.
    Remark: the tweaks in v189 do not affect compression speed.
    Maybe the problem was that I overloaded the CPU. I've set a 50% task limit on my machine and the CPU is at around 70%. I also recompiled it with -DNATIVECPU=ON since I run paq8px exclusively on my AVX2 CPUs and it works on both AMD and Intel just fine.

    Just hoping I don't get into the 100% extract issue I've been before. I'm not really sure if that was caused because of that compilation flag. Will report how it goes.

  20. #2026
    Member
    Join Date
    Feb 2016
    Location
    Luxembourg
    Posts
    546
    Thanks
    203
    Thanked 796 Times in 322 Posts

    paq8px_v190

    Code:
    Changes:
    - New option switch "r" to perform initial retraining of the LSTM on text blocks
    - Support for DEC Alpha executable compression, with a specific transform and model
    As requested by Darek, I made a preliminary model for DEC Alpha executable code. Should get us very close to #2 spot on the Silesia Open Source Benchmark.

    Code:
    File: mozilla, from Silesia Corpus, 51.220.480 bytes
    paq8px_v189 -12                   7.559.180 bytes
    paq8px_v189 -12l                  7.225.479 bytes
    
    paq8px_v190 -12                   6.848.090 bytes
    paq8px_v190 -12l                  6.627.595 bytes
    Attached Files Attached Files
    Last edited by mpais; 5th August 2020 at 01:27. Reason: Add results

  21. Thanks (4):

    Darek (4th August 2020),DZgas (7th August 2020),Mike (4th August 2020),moisesmcardona (4th August 2020)

  22. #2027
    Member
    Join Date
    Jun 2009
    Location
    Puerto Rico
    Posts
    251
    Thanks
    138
    Thanked 52 Times in 39 Posts
    Quote Originally Posted by mpais View Post
    Code:
    Changes:
    - New option switch "r" to perform initial retraining of the LSTM on text blocks
    - Support for DEC Alpha executable compression, with a specific transform and model
    As requested by Darek, I made a preliminary model for DEC Alpha executable code. Should get us very close to #2 spot on the Silesia Open Source Benchmark, early testing points to about 6.629.xxx bytes for mozilla (I'm currently running a test with this final version, should take about 9h)
    No executable?

  23. #2028
    Member
    Join Date
    Feb 2016
    Location
    Luxembourg
    Posts
    546
    Thanks
    203
    Thanked 796 Times in 322 Posts
    Quote Originally Posted by moisesmcardona View Post
    No executable?
    I could've sworn I uploaded it even before writing the post text Oh well, at least the code tags are working again

  24. #2029
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,217
    Thanks
    743
    Thanked 495 Times in 383 Posts
    Quote Originally Posted by mpais View Post
    Code:
    Changes:
    - New option switch "r" to perform initial retraining of the LSTM on text blocks
    - Support for DEC Alpha executable compression, with a specific transform and model
    As requested by Darek, I made a preliminary model for DEC Alpha executable code. Should get us very close to #2 spot on the Silesia Open Source Benchmark, early testing points to about 6.629.xxx bytes for mozilla (I'm currently running a test with this final version, should take about 9h)
    As I found the "english.exp" file was changed - now it's about 2KB smaller. Other two "english" looks as the same.

  25. #2030
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,217
    Thanks
    743
    Thanked 495 Times in 383 Posts
    enwik9 scores of last three changes:

    130'076'196 - enwik9_1423.drt -9eta by Paq8px_v183fix1,
    124'786'260 - enwik9_1423.drt -12eta by Paq8px_v187fix2, change: -4,07%
    121'056'858 - enwik9_1423.drt -12leta by Paq8px_v189, change: -2,99%, time 330'229,11s - it's quite close to cross 120'000'000 line

  26. #2031
    Member
    Join Date
    Jun 2009
    Location
    Puerto Rico
    Posts
    251
    Thanks
    138
    Thanked 52 Times in 39 Posts
    BTW, I noticed when using the `-v` flag it would print levels -1 to -9. I updated it to show instead up to level -12 rather than -9. Be sure to update the code! This is simply a cosmetic change.

  27. #2032
    Member
    Join Date
    Sep 2014
    Location
    Italy
    Posts
    94
    Thanks
    102
    Thanked 39 Times in 25 Posts
    Hi Mpais, what about a preliminare model for mp3 files?
    Do you think should be an easy task?
    Luca

  28. #2033
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,217
    Thanks
    743
    Thanked 495 Times in 383 Posts
    Scores of my testset for paq8px_v190. Option -r gives some slight improvements to textual files.
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	paq8px_v190.jpg 
Views:	32 
Size:	854.3 KB 
ID:	7846  

  29. Thanks:

    LucaBiondi (9th August 2020)

  30. #2034
    Member
    Join Date
    Jun 2009
    Location
    Puerto Rico
    Posts
    251
    Thanks
    138
    Thanked 52 Times in 39 Posts
    Regarding the Intel vs AMD tests, the Intel CPUs are faster than the AMD. And here's this, I compiled it on my AMD Ryzen using NATIVECPU and the Intel machine was faster. Also, consider the fact that the Intel CPUs I have are all i7 mobile CPU parts (4c/8t) while the AMD Ryzen CPUs are all Desktop parts.

    So in conclusion:
    1. Compiled PAQ8PX on my AMD machine using GCC with NATIVECPU
    2. Ran PAQ8PX on Intel and AMD machines, which half threads used.
    3. Intel tasks finished faster than the AMD tasks.

    Update: The first-gen Ryzen is half slower than the 3rd-gen. Mainly due to its AVX2 implementation being just 128-bit and requiring 2 cycles when the 3rd-gen supports 256-bit natively.
    Last edited by moisesmcardona; 10th August 2020 at 19:16.

  31. #2035
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,217
    Thanks
    743
    Thanked 495 Times in 383 Posts
    Quote Originally Posted by moisesmcardona View Post
    Regarding the Intel vs AMD tests, the Intel CPUs are faster than the AMD. And here's this, I compiled it on my AMD Ryzen using NATIVECPU and the Intel machine was faster. Also, consider the fact that the Intel CPUs I have are all i7 mobile CPU parts (4c/8t) while the AMD Ryzen CPUs are all Desktop parts.

    So in conclusion:
    1. Compiled PAQ8PX on my AMD machine using GCC with NATIVECPU
    2. Ran PAQ8PX on Intel and AMD machines, which half threads used.
    3. Intel tasks finished faster than the AMD tasks.

    Update: The first-gen Ryzen is half slower than the 3rd-gen. Mainly due to its AVX2 implementation being just 128-bit and requiring 2 cycles when the 3rd-gen supports 256-bit natively.
    Could you provide some numbers?

  32. #2036
    Member
    Join Date
    Jun 2009
    Location
    Puerto Rico
    Posts
    251
    Thanks
    138
    Thanked 52 Times in 39 Posts
    The AMD tasks are still running. I'll post the CPU runtime once they finish. May take another day for the 3950x system and like 3 more days for the 1700 CPU.

  33. #2037
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,217
    Thanks
    743
    Thanked 495 Times in 383 Posts
    I can test some file to compare Intel CPU to AMD. I'm wondering how much faster is 3950x than 8950HK.

  34. #2038
    Member
    Join Date
    Jun 2009
    Location
    Puerto Rico
    Posts
    251
    Thanks
    138
    Thanked 52 Times in 39 Posts
    Quote Originally Posted by Darek View Post
    I can test some file to compare Intel CPU to AMD. I'm wondering how much faster is 3950x than 8950HK.
    My CPUs are Intel i7-4700MQ and i7-7700HQ. Both finished faster than the AMD CPUs that are currently running.

    Here are some numbers:





    NOTE: Computer names are wrong because I swapped the HDD and I haven't renamed them. But you can see the CPU details there.

    Files being compressed are around 230MB, 24-bit TIFF files. Compressing using -9l.

  35. Thanks:

    Darek (11th August 2020)

  36. #2039
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,217
    Thanks
    743
    Thanked 495 Times in 383 Posts
    @moisesmcardona - yes, it really strange that CPU with 150-160 points in Cinebench r15 (single) runs faster than 3950x which achieve about 200-210 pts in Cinebench.

    My scores of 4 corpuses - both versions -> with -r and w/o this option plus best of them. paq8px_v190 got 2'nd place in Silesia benchmark now! It loses to cmix v18 "only" 183KB.
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	paq8px_v190_4_Corpuses.jpg 
Views:	22 
Size:	2.87 MB 
ID:	7851  

  37. #2040
    Member
    Join Date
    Jun 2009
    Location
    Puerto Rico
    Posts
    251
    Thanks
    138
    Thanked 52 Times in 39 Posts
    Ok, here's the results of the AMD 3950x CPU along with the previous ones from Intel. Took aprox. 200,000 seconds more on AMD than on Intel.




    NOTE: The real number is the CPU time, since that logs the time the process was using the CPU. Run time is the overall time it took for the process to run.

  38. Thanks:

    Darek (11th August 2020)

Page 68 of 71 FirstFirst ... 18586667686970 ... LastLast

Similar Threads

  1. FrontPAQ - GUI frontend for PAQ8PF and PAQ8PX
    By LovePimple in forum Download Area
    Replies: 26
    Last Post: 17th January 2019, 13:36
  2. Alternative paq8px builds
    By M4ST3R in forum Download Area
    Replies: 20
    Last Post: 25th June 2010, 16:19
  3. Optimized paq7asm.asm code not compatible with paq8px?
    By M4ST3R in forum Data Compression
    Replies: 7
    Last Post: 3rd June 2009, 15:34

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •