Page 33 of 34 FirstFirst ... 2331323334 LastLast
Results 961 to 990 of 994

Thread: Paq8pxd dict

  1. #961
    Member
    Join Date
    Sep 2014
    Location
    Italy
    Posts
    102
    Thanks
    110
    Thanked 47 Times in 29 Posts
    Yes no changes for enwik8.
    We reached the max with o40 m3360
    Instead we should see improved compression for enwik9.
    At least i hope!
    Thank you
    Luca

  2. #962
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,237
    Thanks
    762
    Thanked 505 Times in 391 Posts
    ewnik scores:

    15'654'147 - enwik8 -x15 -w -e1,english.dic by paq8pxd_v89_40_3360, change: -0,01%
    123'013'220 - enwik9 -x15 -w -e1,english.dic by paq8pxd_v89_40_3360, change: -0,23%, memory used 32'130MB (by paq8pxd)

    15'654'151 - enwik8 -x15 -w -e1,english.dic by paq8pxd_v89_60_4095, change: 0,00%
    122'945'119 - enwik9 -x15 -w -e1,english.dic by paq8pxd_v89_60_4095, change: -0,06%, memory used 34'335MB (by paq8pxd) - and finally there is a gain about 70kB!
    Last edited by Darek; 19th July 2020 at 18:02.

  3. Thanks:

    LucaBiondi (18th July 2020)

  4. #963
    Member
    Join Date
    Sep 2014
    Location
    Italy
    Posts
    102
    Thanks
    110
    Thanked 47 Times in 29 Posts
    Thank you very much Darek!

    122.945.199 well is not bad at all

    My goal was to push ppm_mod to the limit.
    I have tried a few parameters, As soon as I have a moment I write what I felt.If kaitz or someone wants to apply these small changes to paq8pxd, I will be happy .

    Luca

  5. #964
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,237
    Thanks
    762
    Thanked 505 Times in 391 Posts
    Gain is always gain
    This changes plus maybe LSTM could give us 121'xxx'xxx bytest at the end.

  6. #965
    Member
    Join Date
    Sep 2014
    Location
    Italy
    Posts
    102
    Thanks
    110
    Thanked 47 Times in 29 Posts
    Hi Darek!
    I think the same!!
    What do you think, what is globally the best version between 40 3360 and 60 4095?
    Luca

  7. #966
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,237
    Thanks
    762
    Thanked 505 Times in 391 Posts
    For my testset best version is 40 3360, however as it was visible on enwik9, for big files better is 60 4095.
    I'll need to test 4 corpuses on 60 4095 and then maybe this choise will be easier.

  8. #967
    Member
    Join Date
    Sep 2014
    Location
    Italy
    Posts
    102
    Thanks
    110
    Thanked 47 Times in 29 Posts
    Good idea!
    Luca

  9. #968
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,237
    Thanks
    762
    Thanked 505 Times in 391 Posts
    Scores of 4 Corpuses for paq8pxd_v89_ppm_60_4095.

    Looks like for smaller files tests (Calgary, Canterbury, Maximum Compression and my testset) this version generates worse scores than 40_3360 but for bigger files tests like Silesia and enwik9 scores are better...
    Then it's hard to be objective and say definetely but if we add all 5 corpuses together then 60_4095 wins and wins also for enwik9 then let it be - for my opinion 60_4095 should be best version.
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	paq8pxd_v89_60_4095_4_Corpuses.jpg 
Views:	57 
Size:	2.84 MB 
ID:	7798  

  10. #969
    Member
    Join Date
    Sep 2014
    Location
    Italy
    Posts
    102
    Thanks
    110
    Thanked 47 Times in 29 Posts
    Hi Darek!
    Good job and thank you!
    If Kaitz or Shelwien want adopt these parameters i will be happy!
    ‚ÄčLuca

  11. #970
    Member
    Join Date
    Sep 2015
    Location
    Italy
    Posts
    279
    Thanks
    117
    Thanked 161 Times in 118 Posts
    Quote Originally Posted by kaitz View Post
    https://encode.su/threads/1464-Paq8p...ll=1#post61439
    Just uncomment some LSTM related lines and it works.
    It's not that easy, at least for me.
    I tried to rewrite lstm.inc with the current version of LSTM present in cmix (with the differences needed to use it in paq8pxd), however the decompressed file is different from the original.
    Then I started from the original lstm.inc, but I had the same problem, trying it with paq8pxd 73 (first version in which the management of lstm.inc is present) and 89 (latest version), g++ 6.3 and 7.1 with various options.
    After debugging for some time, it is still not clear to me if it is a compilation problem (but I have tried 2 g++ and with different options) or in the LSTM source (it doesn't seem, but it is difficult to debug).
    Has anyone else tried to enable the LSTM part?
    It's not my current main work, if I can't solve it without taking too much more time I'll have to give up.

  12. #971
    Member
    Join Date
    Aug 2014
    Location
    Argentina
    Posts
    543
    Thanks
    239
    Thanked 94 Times in 74 Posts
    @kaitz: I've found a problem with SZDD preprocessing (apparently) - GitHub issue.

    This file does not make a sound roundtrip. No crashes, but the decompressed file is different to the original. I didn't actually compress the file (-s0) so I guess it's safe to assume the SZDD implementation is the cause.

    By the way, great work with the preprocessor! I wonder if it could be separated into a standalone library to include it on other software, like precomp. Especially since paq8pxd is GPL so its code can't really be shared on most other projects with less restrictive licenses.
    Attached Files Attached Files
    Last edited by Gonzalo; 11th October 2020 at 17:49. Reason: Replaced attachment. See GitHub issue for an explanation.

  13. Thanks:

    kaitz (24th October 2020)

  14. #972
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    514
    Thanks
    211
    Thanked 353 Times in 188 Posts
    Quote Originally Posted by Mauro Vezzosi View Post
    It's not that easy, at least for me.
    I tried to rewrite lstm.inc with the current version of LSTM present in cmix (with the differences needed to use it in paq8pxd), however the decompressed file is different from the original.
    Confirmed. I made GitHub issue for it. Will look into it in january.
    ---
    Also uploaded paq8pxd_v90, if someone wants to test. Cant upload large files now, so source only.
    KZo


  15. Thanks (2):

    Gotty (31st October 2020),Mauro Vezzosi (24th October 2020)

  16. #973
    Member
    Join Date
    Jun 2009
    Location
    Puerto Rico
    Posts
    260
    Thanks
    148
    Thanked 56 Times in 42 Posts
    Quote Originally Posted by kaitz View Post
    Confirmed. I made GitHub issue for it. Will look into it in january.
    ---
    Also uploaded paq8pxd_v90, if someone wants to test. Cant upload large files now, so source only.
    Hey @kaitz, I went ahead and created a CMakeLists.txt file to allow compilation using CMake/Make. See my PR here: https://github.com/kaitz/paq8pxd/pull/12

  17. #974
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,237
    Thanks
    762
    Thanked 505 Times in 391 Posts
    Quote Originally Posted by moisesmcardona View Post
    Hey @kaitz, I went ahead and created a CMakeLists.txt file to allow compilation using CMake/Make. See my PR here: https://github.com/kaitz/paq8pxd/pull/12
    If you can make build with paq8pxd v90 I could test it.
    Question - 4 corpuses are enough to test or I could take another files?

  18. #975
    Member
    Join Date
    Jun 2009
    Location
    Puerto Rico
    Posts
    260
    Thanks
    148
    Thanked 56 Times in 42 Posts
    Quote Originally Posted by Darek View Post
    If you can make build with paq8pxd v90 I could test it.
    Question - 4 corpuses are enough to test or I could take another files?
    Yes sir, attached are both native (AVX2) and standard (SSE2) builds, compiled with Multithreading support (-DMT)
    Attached Files Attached Files

  19. Thanks:

    Darek (25th October 2020)

  20. #976
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,237
    Thanks
    762
    Thanked 505 Times in 391 Posts
    At first my testset for paq8px_v90. Nice improvements. Still some bytes behind latest paq8px (about 100KB - without LSTM) but almost all files got some gains.
    There is only some loses for biggest 24bpp images - both of them loses about 400 bytes.
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	paq8pxd_v90.jpg 
Views:	23 
Size:	855.6 KB 
ID:	8021  

  21. Thanks:

    Gotty (31st October 2020)

  22. #977
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    453
    Thanks
    58
    Thanked 81 Times in 62 Posts
    Paq8pxd90fix1
    - improve jpeg compression by adding 5 apm and 1 mixer context
    paq8pxdv90 -s8 test.jpg 2187172
    paq8pxdv90fix1 -s8 test.jpg
    2185794
    paq8pxdv90 -s8 a10.jpg 623059
    paq8pxdv90fix1 -s8 a10.jpg
    622691y
    here is source code, binary file and batch script to compile inside the package.
    Attached Files Attached Files

  23. #978
    Member
    Join Date
    Jun 2009
    Location
    Puerto Rico
    Posts
    260
    Thanks
    148
    Thanked 56 Times in 42 Posts
    Quote Originally Posted by suryakandau@yahoo.co.id View Post
    Paq8pxd90fix1
    - improve jpeg compression by adding 5 apm and 1 mixer context
    paq8pxdv90 -s8 test.jpg 2187172
    paq8pxdv90fix1 -s8 test.jpg
    2185794
    paq8pxdv90 -s8 a10.jpg 623059
    paq8pxdv90fix1 -s8 a10.jpg
    622691y
    here is source code, binary file and batch script to compile inside the package.
    Please submit your changes to the paq8pxd repo: https://github.com/kaitz/paq8pxd so that Kaitz can review it.

  24. #979
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    453
    Thanks
    58
    Thanked 81 Times in 62 Posts
    Quote Originally Posted by moisesmcardona View Post
    Please submit your changes to the paq8pxd repo: https://github.com/kaitz/paq8pxd so that Kaitz can review it.
    i have created github account (github.com/skandau/paq8pxd) so what is the next step ?

  25. #980
    Member Gotty's Avatar
    Join Date
    Oct 2017
    Location
    Switzerland
    Posts
    620
    Thanks
    372
    Thanked 388 Times in 209 Posts
    You can learn git yourself, the internet is full of introductions, guides, forums. Please make some efforts. Why don't you?

    https://encode.su/threads/342-paq8px...ll=1#post67129
    https://encode.su/threads/342-paq8px...ll=1#post67131
    https://encode.su/threads/342-paq8px...ll=1#post67147

  26. #981
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    453
    Thanks
    58
    Thanked 81 Times in 62 Posts

    Question

    Quote Originally Posted by Gotty View Post
    You can learn git yourself, the internet is full of introductions, guides, forums. Please make some efforts. Why don't you?

    https://encode.su/threads/342-paq8px...ll=1#post67129
    https://encode.su/threads/342-paq8px...ll=1#post67131
    https://encode.su/threads/342-paq8px...ll=1#post67147
    I have tried it but still can't make pull request

  27. #982
    Member Gotty's Avatar
    Join Date
    Oct 2017
    Location
    Switzerland
    Posts
    620
    Thanks
    372
    Thanked 388 Times in 209 Posts
    You did not fork.
    You will need to learn how to drive before you would like to sit in an actual car. Learn git.
    Create git repositories in your account, and do forks, do pull requests, do merges between them until you understand it wholly.

  28. #983
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,237
    Thanks
    762
    Thanked 505 Times in 391 Posts
    Here are scores of paq8xd_v90 for 4 Corpuses. Nice gains for all corpuses, especially about 240KB in Silesia due to big improvement on Mozilla file.
    Unfortunatelly 3 files from MaximumCompression crashes during compression with -x15 option => A10.JPG, ohs.doc and maximumCompression.tar.
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	paq8pxd_v90_4_Corpuses.jpg 
Views:	15 
Size:	2.89 MB 
ID:	8034  

  29. Thanks:

    Gotty (2nd November 2020)

  30. #984
    Member Gotty's Avatar
    Join Date
    Oct 2017
    Location
    Switzerland
    Posts
    620
    Thanks
    372
    Thanked 388 Times in 209 Posts
    Kaitz,
    Line 3599:
     int p1 = state?Maps8b[i]->p(state,m.x.y):0;
    ->
     int p1 = state?Maps8b[i]->p(state,m.x.y):2048;
    ?

  31. #985
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    453
    Thanks
    58
    Thanked 81 Times in 62 Posts
    Quote Originally Posted by Darek View Post
    Here are scores of paq8xd_v90 for 4 Corpuses. Nice gains for all corpuses, especially about 240KB in Silesia due to big improvement on Mozilla file.
    Unfortunatelly 3 files from MaximumCompression crashes during compression with -x15 option => A10.JPG, ohs.doc and maximumCompression.tar.
    Hhmmm thatis strange...using -s8 A10.jpg get 622691 but using -x15 the result is 627083...

  32. #986
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    453
    Thanks
    58
    Thanked 81 Times in 62 Posts
    How about enwik9 result using -x15 -w -e1,English.dic ?

  33. #987
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,237
    Thanks
    762
    Thanked 505 Times in 391 Posts
    Quote Originally Posted by suryakandau@yahoo.co.id View Post
    How about enwik9 result using -x15 -w -e1,English.dic ?
    I'll test it. My estimate is about 100KB less.

    122'945'119 - enwik9 -x15 -w -e1,english.dic by paq8pxd_v89_60_4095, change: -0,06%
    122'838'xxx - enwik9 -x15 -w -e1,english.dic - estimate for paq8px_v90

  34. #988
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    514
    Thanks
    211
    Thanked 353 Times in 188 Posts
    Its ok, p1 is used only state!=0. Should be inside if statment.
    Jpeg error is my mistake. Old mistake i keep making for larger levels.
    Enwik9 may not be better from _60_4095, no changes from that version. Probably no impovment.
    KZo


  35. #989
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,237
    Thanks
    762
    Thanked 505 Times in 391 Posts
    Quote Originally Posted by kaitz View Post
    Its ok, p1 is used only state!=0. Should be inside if statment.
    Jpeg error is my mistake. Old mistake i keep making for larger levels.
    Enwik9 may not be better from _60_4095, no changes from that version. Probably no impovment.
    I've estimate eneik9 score due to slightly better performance for textual files in my testset. There is a some, very little improvement. But for enwik9 it could have visible gain

  36. #990
    Member
    Join Date
    Aug 2008
    Location
    NZ
    Posts
    61
    Thanks
    35
    Thanked 11 Times in 7 Posts

    Question

    Doing some compression testing using paq8pxd V90 (SSE compile; can't use AVX2 compile).

    A pdf file being compressed generates a lot of "Transform fails at 0, skipping . . . " messages when compressing this file!

    paq8pxd v90 does compress the pdf file in the end, but I suspect compression would be better if paq8pxd v90 didn't "fail" on reading/interpreting (parts of) the pdf file(?)

    File attached (7z format compressed).

    Can anyone check this and maybe provide a solution to enable full compression of this file, please?
    Attached Files Attached Files

Page 33 of 34 FirstFirst ... 2331323334 LastLast

Similar Threads

  1. FreeArc compression suite (4x4, Tornado, REP, Delta, Dict...)
    By Bulat Ziganshin in forum Data Compression
    Replies: 554
    Last Post: 26th September 2018, 03:41
  2. Dict preprocessor
    By pat357 in forum Data Compression
    Replies: 5
    Last Post: 2nd May 2014, 22:51

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •