Page 73 of 76 FirstFirst ... 23637172737475 ... LastLast
Results 2,161 to 2,190 of 2253

Thread: paq8px

  1. #2161
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    453
    Thanks
    58
    Thanked 81 Times in 62 Posts
    Quote Originally Posted by Gotty View Post
    Code:
    Fixes and improvements in jpeg model
    - fixed indexing (thank you, mpais!)
    - removed 2 contexts and 1 apm, added 1 context
    - hashing instead of direct contexts in apms (thank you, Kaitz!)
    - using abs for advPred[] and lcp[] in apms (thank you, Kaitz!)
    - reduced memory use in apms (-24 MB on level 8)
    - tweaks
    It looks similar like I do, random forest too...

  2. #2162
    Member Gotty's Avatar
    Join Date
    Oct 2017
    Location
    Switzerland
    Posts
    620
    Thanks
    372
    Thanked 388 Times in 209 Posts
    Yes. Almost.

    The "tweaks" in the last line are tweaks indeed. That is: parameter tuning = trying out different parameters and test them. In that regard it's the same as yours.
    However parameter tuning should be applied to parameters only. Your code contained changes of values (like 1024 to 2048 in m1->set) that are not tunable parameters = they must stay intact when the range of the context is unmodified (which is 1024 in this example).
    Knowing some background helps in parameter tweaking: you must know what you may tune and what shall stay fixed or be calculated.

    It's also important to know how to combine contexts. (Learning opportunity for you. ->)
    When it's about combining vectors, you may add them, average them, weight them. But when the contexts are simply independent values you should not add them.
    Your code (wrong):
    cxt[i++] = hash(++n, hc, advPred[1] / 11 + (runPred[1]), sSum2 >> 6);

    Fix:
    cxt[i++] = hash(++n, hc, advPred[1] / 11, runPred[1], sSum2 >> 6);


    So this release is about fixing your code after I tested the added contexts more thoroughly. I removed some contexts of yours because they seemed to have no benefit. My question is: why did you add them? Did you have some specific files where they are beneficial?
    Last edited by Gotty; 27th October 2020 at 01:57. Reason: language

  3. #2163
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    453
    Thanks
    58
    Thanked 81 Times in 62 Posts
    Quote Originally Posted by Gotty View Post
    Yes. Almost.

    The "tweeks" are tweaks. Parameter tuning = trying out different parameters and test them. In that part it's the same as yours.
    However parameter tuning should be applied to parameters only. Your code contained changes to values that are not parameters (like those 1024->2048 changes), and so should be exactly some exact value.
    So it helps parameter tweaking when you know some background: what you may tune and what shall stay fixed or be calculated.

    It's also important to know how you can combine contexts. (Learning opportunity for you. ->)
    When it's about combining vectors, you may add them, average them, weight them. But when the contexts are simply independent values you should not add them.
    Your code (wrong):
    cxt[i++] = hash(++n, hc, advPred[1] / 11 + (runPred[1]), sSum2 >> 6);

    Fix:
    cxt[i++] = hash(++n, hc, advPred[1] / 11, runPred[1], sSum2 >> 6);


    So this release is mostly fixing your code and testing your additional contexts more thoroughly. I removed some contexts of yours because they seemed to not have any benefit. My question is: why did you add them? Did you have some files where they are beneficial?
    When I add that context it can reduce test.jpg size

  4. #2164
    Member Gotty's Avatar
    Join Date
    Oct 2017
    Location
    Switzerland
    Posts
    620
    Thanks
    372
    Thanked 388 Times in 209 Posts
    I see.
    You should test your changes with many different files. We don''t build paq8px to compress a single test.jpg. It's a general compressor.
    Never add complexity or use more memory or slow down compression just for one file.
    If you don't test with multiple files, you will not be able to properly evaluate your changes. So please do.

  5. #2165
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    453
    Thanks
    58
    Thanked 81 Times in 62 Posts
    Quote Originally Posted by Gotty View Post
    I see.
    You should test your changes with many different files. We don''t build paq8px to compress a single test.jpg. It's a general compressor.
    Never add complexity or use more memory or slow down compression just for one file.
    If you don't test with multiple files, you will not be able to properly evaluate your changes. So please do.

    May you give some JPEG files ? I don't have any files except test.jpg

  6. #2166
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,237
    Thanks
    762
    Thanked 505 Times in 391 Posts
    Quote Originally Posted by suryakandau@yahoo.co.id View Post
    May you give some JPEG files ? I don't have any files except test.jpg
    Here are two files: A10.jpg from Maximum Compression and F.JPG file from my testset.
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	F.JPG 
Views:	33 
Size:	109.4 KB 
ID:	8024   Click image for larger version. 

Name:	A10.jpg 
Views:	31 
Size:	822.7 KB 
ID:	8025  

  7. #2167
    Member Gotty's Avatar
    Join Date
    Oct 2017
    Location
    Switzerland
    Posts
    620
    Thanks
    372
    Thanked 388 Times in 209 Posts
    These are the ones that cover some special cases:

    MJPEG

    1)
    http://jjc.freeshell.org/turning_pages.html
    Turning Pages AVI

    2)
    https://www.mainconcept.com/getting-...d/samples.html
    Motion-JPEG, 1920x1080

    3)
    http://www.doicamera.com/digital_video.htm
    Mjpeg sample video (22 mb)

    JPEGs with thumbnails
    https://encode.su/threads/342-paq8px...ll=1#post64174
    https://encode.su/threads/342-paq8px...ll=1#post64175
    https://encode.su/threads/342-paq8px...ll=1#post64176

    Several multipage TIFF with JPEGs
    http://156.35.94.8/tmp/

    For plain single JPEG images you'll just need to look around in your pc or google. You'll need to find baseline JPEG files as paq8px does not support progressive JPEGs.

  8. #2168
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    453
    Thanks
    58
    Thanked 81 Times in 62 Posts
    Paq8px197
    - improve jpeg recompression

    Size Compressed Sec
    paq8pxd_v90 -s8 PH01046J.JPG 135611 107464 6
    paq8px -8 PH01046J.JPG 135611 105624 7
    paq8pxd_v90 -s8 20200708_105932.jpg 112038 81483
    paq8px -8 f.jpg (darek corpus) 112038 81299
    paq8pxd_v90 -s8 A10.jpg 842468 623059 25
    paq8px -8 A10.jpg 842468 624267 36
    paq8pxd_v90 -s8 test.JPG 3162196 2187172 85
    paq8px -8 test.JPG 3162196 2194077 134
    hhmm looks paq8px better than paq8pxd on small jpeg files...
    Attached Files Attached Files

  9. #2169
    Member Gotty's Avatar
    Join Date
    Oct 2017
    Location
    Switzerland
    Posts
    620
    Thanks
    372
    Thanked 388 Times in 209 Posts
    Could you please withdraw this version? Probably delete the attachment.
    Could you follow our guidelines, please?
    That is: 1) use git. 2) before posting any tweaks do understand how paq8px works.

  10. #2170
    Member Gotty's Avatar
    Join Date
    Oct 2017
    Location
    Switzerland
    Posts
    620
    Thanks
    372
    Thanked 388 Times in 209 Posts
    I promised that we will help if you have any questions. So let's do it the other way around. I'll ask.
    Let's look at this single line, which is new:
    Code:
     m.set( coef | min(5+2*(!comp),zu+zv),1024);
    What does 1024 mean on this line?
    What the "|" operator does exactly?
    What is the range of coef?
    What is the range of 5+2*(!comp)?
    What is the range of zu and zv?
    What do you need to adjust if you add a new mixer context?

  11. Thanks:

    Mike (29th October 2020)

  12. #2171
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    453
    Thanks
    58
    Thanked 81 Times in 62 Posts
    Quote Originally Posted by Gotty View Post
    Could you please withdraw this version? Probably delete the attachment.
    Could you follow our guidelines, please?
    That is: 1) use git. 2) before posting any tweaks do understand how paq8px works.

    What is your email ? Maybe it is more ok to discuss with email...if I use git I think it more universalthan use this forum...

  13. #2172
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    453
    Thanks
    58
    Thanked 81 Times in 62 Posts
    Quote Originally Posted by Gotty View Post
    I promised that we will help if you have any questions. So let's do it the other way around. I'll ask.
    Let's look at this single line, which is new:
    Code:
     m.set( coef | min(5+2*(!comp),zu+zv),1024);
    What does 1024 mean on this line?
    What the "|" operator does exactly?
    What is the range of coef?
    What is the range of 5+2*(!comp)?
    What is the range of zu and zv?
    What do you need to adjust if you add a new mixer context?
    Please give some explanation about it...

  14. #2173
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    453
    Thanks
    58
    Thanked 81 Times in 62 Posts
    Quote Originally Posted by suryakandau@yahoo.co.id View Post
    Please give some explanation about it...

    what is the use of new mixer context ? what is the use of new apm ?

  15. #2174
    Member Gotty's Avatar
    Join Date
    Oct 2017
    Location
    Switzerland
    Posts
    620
    Thanks
    372
    Thanked 388 Times in 209 Posts
    How about doing some homework? That is very-very essential. You need to do efforts, please.

  16. #2175
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    453
    Thanks
    58
    Thanked 81 Times in 62 Posts
    Quote Originally Posted by suryakandau@yahoo.co.id View Post
    what is the use of new mixer context ? what is the use of new apm ?
    so what is your explanation about them ?

  17. #2176
    Member Gotty's Avatar
    Join Date
    Oct 2017
    Location
    Switzerland
    Posts
    620
    Thanks
    372
    Thanked 388 Times in 209 Posts
    I don't understand your question.
    Are you asking me about your new contexts or new contexts in general?

  18. #2177
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    453
    Thanks
    58
    Thanked 81 Times in 62 Posts
    Quote Originally Posted by Gotty View Post
    I don't understand your question.
    Are you asking me about your new contexts or new contexts in general?
    How should be my new context mixer ?

  19. #2178
    Member Gotty's Avatar
    Join Date
    Oct 2017
    Location
    Switzerland
    Posts
    620
    Thanks
    372
    Thanked 388 Times in 209 Posts
    I'm still unsure what you mean. Are you asking me how to fix this code?
    Code:
     m.set( coef | min(5+2*(!comp),zu+zv),1024);

  20. #2179
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    453
    Thanks
    58
    Thanked 81 Times in 62 Posts
    Quote Originally Posted by Gotty View Post
    I'm still unsure what you mean. Are you asking me how to fix this code?
    Code:
     m.set( coef | min(5+2*(!comp),zu+zv),1024);
    Yes..thank you..

  21. #2180
    Member Gotty's Avatar
    Join Date
    Oct 2017
    Location
    Switzerland
    Posts
    620
    Thanks
    372
    Thanked 388 Times in 209 Posts
    Aha, I see.
    Here it is.
     m.set(coef | min(5+2*(!comp),zu+zv)<<8, 2048); 

    You will be able to fix it yourself as soon as you do the homework.
    When doing development however, you'll need to validate any changes: for example this mixer context could be too large and thus waste memory or could be disadvantageous for some type of images. I did not test that.
    Fixing this line is not enough though. You have more problems.
    You don't need to rush with the fixing. It will not work on the long run if you expect me (or probably anyone else) to validate your changes, fix the code, run tests, merge, do a pull request.
    Please do your homework. You should not try anything until you can answer (at least!!!) those questions above.

  22. #2181
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    453
    Thanks
    58
    Thanked 81 Times in 62 Posts
    Quote Originally Posted by Gotty View Post
    Aha, I see.
    Here it is.
     m.set(coef | min(5+2*(!comp),zu+zv)<<8, 2048); 

    You will be able to fix it yourself as soon as you do the homework.
    When doing development however, you'll need to validate any changes: for example this mixer context could be too large and thus waste memory or could be disadvantageous for some type of images. I did not test that.
    Fixing this line is not enough though. You have more problems.
    You don't need to rush with the fixing. It will not work on the long run if you expect me (or probably anyone else) to validate your changes, fix the code, run tests, merge, do a pull request.
    Please do your homework. You should not try anything until you can answer (at least!!!) those questions above.
    Why min(5+2*(!comp),zu+zv) is left shifted by 8 ? Not by 4 only ?

  23. #2182
    Member Gotty's Avatar
    Join Date
    Oct 2017
    Location
    Switzerland
    Posts
    620
    Thanks
    372
    Thanked 388 Times in 209 Posts
    These are my questions that help you answering your question:

    What is the range of coef?
    What is the range of 5+2*(!comp)?
    What is the range of zu and zv?

    Can you answer them? (I believe, you can.)

  24. #2183
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    453
    Thanks
    58
    Thanked 81 Times in 62 Posts
    Quote Originally Posted by Gotty View Post
    These are my questions that help you answering your question:

    What is the range of coef?
    What is the range of 5+2*(!comp)?
    What is the range of zu and zv?

    Can you answer them? (I believe, you can.)
    If iam not wrong the range of coef is 8bit

  25. #2184
    Member Gotty's Avatar
    Join Date
    Oct 2017
    Location
    Switzerland
    Posts
    620
    Thanks
    372
    Thanked 388 Times in 209 Posts
    Correct.
    ​What about the others?

  26. #2185
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    453
    Thanks
    58
    Thanked 81 Times in 62 Posts
    Quote Originally Posted by Gotty View Post
    Correct.
    ​What about the others?
    Maybe 1 bit

  27. #2186
    Member Gotty's Avatar
    Join Date
    Oct 2017
    Location
    Switzerland
    Posts
    620
    Thanks
    372
    Thanked 388 Times in 209 Posts
    No.

    I had hope but was mistaken. Those questions are not difficult at all. So the situation is worse.
    Your todo list has just grown (in this priority order):

    Learn about binary representation and binary arithmetic
    Learn git
    Learn paq8px

    You'll have to seek help regarding the first 2 items elsewhere, when you are ready you may start learning paq8px. Expect it to be some months' time with intensive learning.
    Please do not post any new versions until you master those areas. I'm serious. Paq8px is no playground.

  28. #2187
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    453
    Thanks
    58
    Thanked 81 Times in 62 Posts
    Quote Originally Posted by Darek View Post
    Some enwik scores (mostly enwik8 ) for the latest paq8px versions:

    16'190'519 - enwik8 -12 by Paq8px_v187fix2
    16'080'588 - enwik8 -12eta by Paq8px_v187fix2
    15'889'931 - enwik8.drt -12eta by Paq8px_v187fix2
    127'626'051 - enwik9_1423 -12eta by Paq8px_v187fix2
    124'786'260 - enwik9_1423.drt -12eta by Paq8px_v187fix2

    15'900'206 - enwik8 -12leta by Paq8px_v188, change: -1,12%
    15'503'221 - enwik8.drt -12leta by Paq8px_v188, change: -2,43%

    15'907'081 - enwik8 -12leta by Paq8px_v188b, change: 0,04%
    15'505'761 - enwik8.drt -12leta by Paq8px_v188b, change: 0,02%

    15'896'588 - enwik8 -12leta by Paq8px_v189, change: -0,07%
    15'490'302 - enwik8.drt -12leta by Paq8px_v189, change: -0,10%
    121'056'858 - enwik9_1423.drt -12leta by Paq8px_v189, change: -2,99% - best score measured

    15'896'196 - enwik8 -12leta by Paq8px_v190, change: 0,00%
    15'896'126 - enwik8 -12lreta by Paq8px_v190, change: 0,00%
    15'490'045 - enwik8.drt -12leta by Paq8px_v190, change: 0,00%
    15'490'045 - enwik8.drt -12lreta by Paq8px_v190, change: 0,00%

    15'888'954 - enwik8 -12lreta by Paq8px_v191, change: -0,05%
    15'638'056 - enwik8.drt -12lreta by Paq8px_v191, change: 0,96%

    15'898'305 - enwik8 -12lreta by Paq8px_v191a, change: 0,01%
    15'638'254 - enwik8.drt -12lreta by Paq8px_v191a, change: 0,96%

    15'898'544 - enwik8 -12lreta by Paq8px_v192, change: 0,00%
    15'888'545 - enwik8 -12leta by Paq8px_v192, change: -0,06%
    15'638'126 - enwik8.drt -12lreta by Paq8px_v192, change: 0,00%
    15'479'471 - enwik8.drt -12leta by Paq8px_v192, change: -1,02%

    15'884'947 - enwik8 -12lreta by Paq8px_v193, change: -0,02% - best score measured
    15'476'230 - enwik8.drt -12lreta by Paq8px_v193, change: -0,02%- best score measured
    126'150'760 - estimated enwik9_1423 -12lreta by Paq8px_v193, change: -0,02% - estimated best score - time to compress about 6 days
    120'946'885 - estimated enwik9_1423.drt -12lreta by Paq8px_v193, change: -0,02% - estimated best score - time to compress about 4 days

    15'899'080 - enwik8 -12lreta by Paq8px_v193fix2, change: 0,09%
    15'477'066 - enwik8.drt -12lreta by Paq8px_v193fix2, change: 0,01%

    Looks that paq8px v193 got the best scores for enwik8 and probably on enwik9, however until the test, the best score of enwik9 belongs to paq8px v189.
    It's close to break 120'000'000 barrier but there is about 1MB to gain
    where can i get enwik8.drt and enwik9.drt ?

  29. #2188
    Member Gotty's Avatar
    Join Date
    Oct 2017
    Location
    Switzerland
    Posts
    620
    Thanks
    372
    Thanked 388 Times in 209 Posts
    drt is a preprocessor: it transforms (text) files so they may be compressed better.
    You'll find such preprocessors here (including drt):
    https://encode.su/threads/3240-Text-ish-preprocessors

  30. #2189
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    453
    Thanks
    58
    Thanked 81 Times in 62 Posts
    Quote Originally Posted by Gotty View Post
    drt is a preprocessor: it transforms (text) files so they may be compressed better.
    You'll find such preprocessors here (including drt):
    https://encode.su/threads/3240-Text-ish-preprocessors
    Thank you. Btw I have tested your edited changes but the result got bigger than before

  31. #2190
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    453
    Thanks
    58
    Thanked 81 Times in 62 Posts
    Quote Originally Posted by Gotty View Post
    Aha, I see.
    Here it is.
     m.set(coef | min(5+2*(!comp),zu+zv)<<8, 2048); 

    You will be able to fix it yourself as soon as you do the homework.
    When doing development however, you'll need to validate any changes: for example this mixer context could be too large and thus waste memory or could be disadvantageous for some type of images. I did not test that.
    Fixing this line is not enough though. You have more problems.
    You don't need to rush with the fixing. It will not work on the long run if you expect me (or probably anyone else) to validate your changes, fix the code, run tests, merge, do a pull request.
    Please do your homework. You should not try anything until you can answer (at least!!!) those questions above.
    I have tested this code on f.jpg (darek corpus) the result is bigger than before.

Page 73 of 76 FirstFirst ... 23637172737475 ... LastLast

Similar Threads

  1. FrontPAQ - GUI frontend for PAQ8PF and PAQ8PX
    By LovePimple in forum Download Area
    Replies: 26
    Last Post: 17th January 2019, 14:36
  2. Alternative paq8px builds
    By M4ST3R in forum Download Area
    Replies: 20
    Last Post: 25th June 2010, 17:19
  3. Optimized paq7asm.asm code not compatible with paq8px?
    By M4ST3R in forum Data Compression
    Replies: 7
    Last Post: 3rd June 2009, 16:34

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •