Page 2 of 4 FirstFirst 1234 LastLast
Results 31 to 60 of 92

Thread: Papa’s Optimizer

  1. #31
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    93
    Thanks
    56
    Thanked 47 Times in 29 Posts
    Quote Originally Posted by CompressMaster View Post
    Krishty, why you uploaded new version in form of external links? It seems to be better to upload each version directly in attachments as I explained earlier.
    But I did! Check the attachment in the first post
    Last edited by Krishty; 30th September 2019 at 17:37.

  2. #32
    Member
    Join Date
    Mar 2016
    Location
    Croatia
    Posts
    195
    Thanks
    82
    Thanked 14 Times in 12 Posts

    problematic gif

    I am trying to downsize larger gif, but Papa's is doing to already 4 hours or so.

    Is there any way to use papa optimizer with less time requirements but still size it down?...or even with lossy compression.

    For testing purposes file is in 7zip in attachment
    Attached Files Attached Files

  3. #33
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    93
    Thanks
    56
    Thanked 47 Times in 29 Posts
    Yeah, GIFs can take months to complete and it makes sense considering that I go for maximal compression. Speeding this up is a TODO and I’m strictly lossless, so no luck right now.

    You could try Nikkho’s File Optimizer, or open a command window in the tools directory and directly call gifsicle -o out.gif in.gif. But your GIF seems pretty well-optimized already, so don’t expect large returns.

  4. #34
    Member
    Join Date
    Jul 2014
    Location
    Mars
    Posts
    206
    Thanks
    142
    Thanked 13 Times in 12 Posts
    I know main goal is max compression, but Fast and Normal options would be great for quick testing / casual usage.

  5. #35
    Member
    Join Date
    Oct 2019
    Location
    Latvia
    Posts
    1
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Hi!
    I tried to optimize an odt file for test. App just sopped responding after extracting content, but at the same time CPU is idling. I'm using Windows 10 build 18362.356

  6. #36
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    93
    Thanks
    56
    Thanked 47 Times in 29 Posts
    Thanks for testing and very sorry for the problems you experience! Can you please help me investigate via the following:
    1. When the app stopped responding, open Task Manager -> Details -> Right-click Best Optimizer.exe -> Create Dump -> send the file to me (info at papas-best.com), maybe zipped.
    2. Clear your %TEMP% directory (C:\Users\ASkript\Appdata\Local\Temp) from any sub-directories starting with "c1c". Start the optimization again. When the app hangs, there should be a new folder starting with "c1c" and it should contain a log.txt file. Please send this file as well.

  7. #37
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    93
    Thanks
    56
    Thanked 47 Times in 29 Posts
    Updated version with drag’n’drop support and optional foreground priority: https://papas-best.com/downloads/opt...20Optimizer.7z

    This was a major UI overhaul, so regard this as a beta release.
    • major UI overhaul (now supporting drag’n’drop)
    • moved options to new General tab
    • fixed 7-Zip settings
    • fixed crashes with analysis
    • fixed a typo
    • fixed tab flickering

  8. Thanks:

    necros (22nd October 2019)

  9. #38
    Member JamesWasil's Avatar
    Join Date
    Dec 2017
    Location
    Arizona
    Posts
    110
    Thanks
    96
    Thanked 19 Times in 18 Posts
    Papa's Optimizer: Better Ingredients. Better Refinements. Papa Op's.

    Once the optimizations can be delivered under 30 minutes with a Papa Tracker, I would tip for that.

  10. Thanks:

    Krishty (14th October 2019)

  11. #39
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    93
    Thanks
    56
    Thanked 47 Times in 29 Posts
    Hah! I’ll see what I can do

  12. Thanks (2):

    JamesWasil (14th October 2019),necros (22nd October 2019)

  13. #40
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    93
    Thanks
    56
    Thanked 47 Times in 29 Posts
    Dear ladies and gentleman,

    another update attached to the first post or downloadable via https://papas-best.com/downloads/opt...20Optimizer.7z .

    It is, first and foremost, a bugfix release:
    • options are now disabled during optimization
    • fixed possible deadlock with Leanify
    • fixed possible deadlock with damaged archives
    • fixed missing 7z.dll
    • fixed launch error message
    • fixed tab glitches
    • fixed keyboard navigation

    @ASkript: Bugs have been fixed that only surfaced on my freshly installed Windows 10. There’s a chance that it works better for you now.

    For those who haven’t seen the reworked UI yet:



    The UI will now flicker with the Windows 7 Classic Theme during resizing, but that’s the price you have to pay for correct tab rendering on Windows 10. (If anyone is interested in Win32 tab control details, I wrote about the changes in German here).

    Please let me know if I broke anything!

  14. Thanks (3):

    dado023 (20th October 2019),JamesWasil (20th October 2019),necros (22nd October 2019)

  15. #41
    Member
    Join Date
    Jul 2014
    Location
    Mars
    Posts
    206
    Thanks
    142
    Thanked 13 Times in 12 Posts
    Considered to upload it to Github?

  16. #42
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    93
    Thanks
    56
    Thanked 47 Times in 29 Posts
    Quote Originally Posted by necros View Post
    Considered to upload it to Github?
    Yes, but only after further code clean-up. When I started the program a few years ago, it was just meant as a replacement for my Batch scripts, with no error handling and no public distribution in mind, and entangled with my other projects. It’ll take a while to resolve that.

    If you want to play around, PM me and I’ll send you a 7z with the Visual Studio project.

    P.S.: First build from 2015; not so much change after all


  17. #43
    Member
    Join Date
    Jul 2014
    Location
    Mars
    Posts
    206
    Thanks
    142
    Thanked 13 Times in 12 Posts
    P.S. where did you get archipng utility? Can`t find source

  18. #44
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    93
    Thanks
    56
    Thanked 47 Times in 29 Posts
    I wrote it myself, in 2009 or so.

    It just runs 32 trials with each filter on or off (2^5) and uses zlib with Z_FIXED compression to determine the smallest result. Then it writes this filter combination without any compression.

    IIRC, Z_FIXED behaves like a pretty good approximation for PPMd; better than the other modes. So you will probably get the best results compressing these PNGs with PPMd afterwards.

    But keep in mind, it has been ten years since I worked on it so I’m happy to learn other tools doing better jobs

    Edit: Attached the only copy of the source code I have.
    Attached Files Attached Files

  19. Thanks:

    maadjordan (26th October 2019)

  20. #45
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    93
    Thanks
    56
    Thanked 47 Times in 29 Posts
    Some development notes in case anyone is interested.

    Data integrity is paramount; there is nothing worse than finding out your backups are corrupted when you desperately need them.

    I want my Optimizer to perform an integrity check on optimized data to find out whether something broke. Since rumors go around that defluff may corrupt archives (though I cannot reproduce this specific case), I decided to start checking the results of ZIP/gzip/Office optimization.

    I imagined I’d call 7-Zip’s checksum feature before and after Deflate optimization and compare the output. Should be simple enough.

    But it’s not. As I noticed quickly, 7-Zip treats archives with just one file in them as a special case – it only computes a checksum of data then, but not of names, to optimize for information size.

    After working around this, checksums failed on almost all ZIP files I tried. The reason is the ZIP central directory after optimization with ECT/defluff/DeflOpt does not contain directory records any more. Directory names are instead coded directly into file names as relative paths (a size optimization, obviously). 7-Zip does not consider this case during checksum computation. I reported this to Igor Pavlov, but I don’t know yet whether this optimization is legal in the first place.

    You’ll probably have to wait for a new version until I worked around all this and tested it thoroughly.

  21. #46
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    93
    Thanks
    56
    Thanked 47 Times in 29 Posts
    I’ve released a new version. See https://papas-best.com/optimizer_en#download or attachments to first post.

    Changes:

    • added checksum test to archive optimization
      I want to avoid silent data corruption at any cost. For ZIP/gzip, the Optimizer now uses 7-Zip to compute checksums of archive content before and after optimization with AdvZIP/ECT/DeflOpt/defluff. If data is corrupted, the job will abort with the message ERROR: content broke during optimization and the original file remains untouched.
      I noticed that ECT does not have any means to disable recursive optimization. I.e. if you optimize a ZIP which contains another ZIP, and you unchecked recursive, then optimization will fail with said message because ECT optimized with the inner ZIP file even though you didn’t want that. I still need to decide how this situation should be treated gracefully.
      Names will not be checked. I would have loved to do this, but:
      • DeflOpt deletes all empty directories from archives, changing the checksum. I could work around that, but … it takes time.
      • 7-Zip has a bug with checksums of file names in ZIPs if they contain relative directories (as commonly produced by DeflOpt/defluff).
      • 7-Zip cannot compute name checksums for single items (as an information size optimization).

    • added setup
      You can now download a MSI package of the optimizer. It’s plain simple; no choosing paths or anything. By default, it installs for the current user only. If you want to install it for all users, run msiexec /i "Papas Best Optimizer.msi" MSIINSTALLPERUSER=""

    • fixed mislabeled Cancel button
      There were two Optimize buttons instead of Optimize and Cancel. Stupid copy-paste error during UI cleanup. Fixed.

    • fixed one thread appearing dead in the UI
      If your CPU had four cores and you selected four threads, the 4th thread would appear stuck even though it optimized fine. Fixed that.

    • fixed recursive ZIP setting
      The recursive check often did not work. Fixed that.

    • fixed gzip optimization potentially deleting filenames
      There was a formatting bug in ECT’s command line, hiding the --strict switch for gzip archives which normally forces preserving file names. My tests did not produce any errors, but it is possible that some gzip files lost file names. Anyway, that’s fixed now.

  22. Thanks (3):

    JamesWasil (16th November 2019),necros (20th December 2019),OneDeltaTenTango (10th November 2019)

  23. #47
    Member
    Join Date
    May 2016
    Location
    Waverly, KS, USA, Earth, Sol System, Milky Way
    Posts
    5
    Thanks
    3
    Thanked 2 Times in 2 Posts
    Did some quick tests. I really like the multi threading.
    Its not APNG safe, test file https://red.dazoe.net/stuff/9.png
    Just to see what happens, I threw my comics/manga at it and will its still Analyzing after +8hr but still progressing. granted its like ~75,000 files.
    On the other-hand FileOptimizer from https://nikkhokkho.sourceforge.io/st...=FileOptimizer would at-least be working on it by now, sadly its not multi threaded and Pingo from https://css-ig.net/ would have been done with it by now if it managed to not hit a folder with a super long path or strange characters.

    If i didn't miss it somewhere would it be possible to add an endless trials/brute-force compression option so it endlessly loops on a batch of pngs.

  24. Thanks:

    Krishty (10th November 2019)

  25. #48
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    93
    Thanks
    56
    Thanked 47 Times in 29 Posts
    Glad you like it!

    Quote Originally Posted by OneDeltaTenTango View Post
    Its not APNG safe, test file https://red.dazoe.net/stuff/9.png
    Excellent find! I’ll see what I can do.

    Just to see what happens, I threw my comics/manga at it and will its still Analyzing after +8hr but still progressing. granted its like ~75,000 files.
    Nonetheless, that’s too long. I recommend:
    1. Throw a dozen of files at it so Optimize becomes enabled and click that. Really just a few files to get it going with all available threads. Then throw your huge directories at it and it will analyze in the background while optimizing. (I may fix that clunky handling in a future version.)
    2. I know Windows has some limitation where you cannot create more than 65k temporary directories. My Optimizer may well be affected. Better run it with batches of 10k until I reworked the temporary directory handling.


    If i didn't miss it somewhere would it be possible to add an endless trials/brute-force compression option so it endlessly loops on a batch of pngs.
    You will notice PNG takes extremely long already, so be careful what you wish for Infinite looping will not buy you anything, I’m afraid. The reason is in the curves here: https://encode.su/threads/2274-ECT-a...ll=1#post61627
    I’m obsessed with squeezing out single bytes, rest assured, but we’re within 100 B of the perfect result using the current tools and it takes other means to improve compression. E.g. I’ll try running huffmix on a collection of ECT passes soon.

  26. Thanks:

    OneDeltaTenTango (11th November 2019)

  27. #49
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    511
    Thanks
    63
    Thanked 95 Times in 74 Posts
    Quote Originally Posted by Krishty View Post
    Glad you like it!

    Excellent find! I’ll see what I can do.

    Nonetheless, that’s too long. I recommend:
    1. Throw a dozen of files at it so Optimize becomes enabled and click that. Really just a few files to get it going with all available threads. Then throw your huge directories at it and it will analyze in the background while optimizing. (I may fix that clunky handling in a future version.)
    2. I know Windows has some limitation where you cannot create more than 65k temporary directories. My Optimizer may well be affected. Better run it with batches of 10k until I reworked the temporary directory handling.


    You will notice PNG takes extremely long already, so be careful what you wish for Infinite looping will not buy you anything, I’m afraid. The reason is in the curves here: https://encode.su/threads/2274-ECT-a...ll=1#post61627
    I’m obsessed with squeezing out single bytes, rest assured, but we’re within 100 B of the perfect result using the current tools and it takes other means to improve compression. E.g. I’ll try running huffmix on a collection of ECT passes soon.
    https://youtu.be/eh4c0prEU7g
    How extremely long for PNG file ? 1 day ? 1 week ?

  28. #50
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    93
    Thanks
    56
    Thanked 47 Times in 29 Posts
    What’s the relation of your link to the topic?

    My CPU is old, so your mileage may vary:

    RGBA 100²: a minute or so
    1000² pixels: 1–2 hours
    for palettized images: multiply with 120 due to ECT trying 120 palette permutations

    At least it scales very well with the number of cores.
    Last edited by Krishty; 11th November 2019 at 12:58.

  29. #51
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    511
    Thanks
    63
    Thanked 95 Times in 74 Posts
    Quote Originally Posted by Krishty View Post
    What’s the relation of your link to the topic?

    My CPU is old, so your mileage may vary:

    RGBA 100²: a minute or so
    1000² pixels: 1–2 hours
    for palettized images: multiply with 120 due to ECT trying 120 palette permutations

    At least it scales very well with the number of cores.
    I just want to show that android can do computer vision not too long. I think computer vision and data compression has the related process. Like what or which byte to recognize etc.

  30. #52
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    93
    Thanks
    56
    Thanked 47 Times in 29 Posts
    Annotated changelog of the latest version from https://papas-best.com/optimizer_en (testing is welcome as always):

    optimization now starts immediately
    No more analyze/optimize separation; should make it easier to use

    fixed gzip inadvertently being converted to ZIP
    Due to an error in a command line, some gzip archives were sometimes converted to ZIP by accident. If you used the previous version to optimize gz archives and you care about the format, please check the optimized files.

    fixed handle exhaustion due to log file leaks
    fixed cleanup of temporary directories

    There was a leak in log file management. It surfaced as seemingly random failures or crashes when optimizing thousands of files and may have spammed your temp directory.

    new multi-threading
    The threading has been completely reworked. The number of threads is now selected in the bottom-right instead via options.
    • You can change the number of threads during optimization. (But consider that lowering it will not take effect immediately because no jobs are being forcefully canceled.)
    • You can pause the optimization by selecting zero threads.
    • Countless scheduling bugs have been fixed, e.g. a DOC optimization expanding to multiple sub-optimizations not being multi-threaded.
    • Optimization starts with the first file you add and runs parallel with further analysis.
    • It’s also a very minimal amount faster because the new implementation has less locking than the old one. (It is now based on a producer-consumer pattern with a thread pool and condition variables.)

    items can now be canceled before optimization started
    cancelation now considers sub-items

    … because that kept annoying me.

    fixed optimized files sometimes being timestamped older than originals
    fixed a crash after optimizing many files

    Wrong math and a null pointer access.

    Sorry that I didn’t get to APNG yet. I have a similar problem with JPG files where some cameras stuff multiple images into the same file, but my Optimizer deletes all but the first. I’m looking into it.

  31. Thanks (2):

    JamesWasil (7th December 2019),necros (20th December 2019)

  32. #53
    Member
    Join Date
    Jul 2014
    Location
    Mars
    Posts
    206
    Thanks
    142
    Thanked 13 Times in 12 Posts
    Can you show example of such jpg which stuff 2 images in single file?

  33. #54
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    93
    Thanks
    56
    Thanked 47 Times in 29 Posts
    Quote Originally Posted by necros View Post
    Can you show example of such jpg which stuff 2 images in single file?
    Sure thing! I attached two samples, both taken by friends of mine.

    One is an MPO renamed to JPG. MPO is Multiple Picture Object and it’s an extension to the JPG standard. It basically stores two JPGs for stereoscopic display. Using my optimizer on this file reduces its size to 49%, mostly so by just deleting the other image, which is very unfortunate. At least the format is well-documented, so with some effort I might be able to handle it properly.

    The other file is something I don’t understand at all. It’s a JPEG taken with a Huawei phone. At 3968×2976 (12 MP), it amounts to 18,2 MiB(!!!). If I run it through my optimizer, it reduces to 3,2 MiB (the expected size). The thing is – I have no idea what’s been deleted. My friend says, it’s a feature to share “moments” instead of just pictures. If anybody could tell me what’s the thing with this file, I’d be glad to know!
    Attached Files Attached Files

  34. #55
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    630
    Thanks
    288
    Thanked 252 Times in 128 Posts
    Quote Originally Posted by Krishty View Post
    Sure thing! I attached two samples, both taken by friends of mine.
    The first one seems to be just a matter of parsing as the MPO format just is the concatenation of two standard JPGs (with metainformation in APP2 markers) - if treated as a file with two embedded JPGs, it should work. Precomp does this and detects the 2 JPG streams successfully:

    Code:
    Possible JPG found at position 0, length 619813
    Possible JPG found at position 619813, length 563230
    
    // note: 619813 + 563230 = 1183043 which is the size of the file
    The second one is JPG data (position 0, length 3677374) followed by some header data (~5 KB). After that, "ftypmp42" indicates a video which can be extracted (beginning at offset 0x3831EA) and played (tested with VLC). So it's basically a short video (2 seconds, 960x720 H264 video, AAC audio) from the moment the picture was taken - attached it to the post. There could be some data after it, too, but shouldn't be too much if any.
    Attached Files Attached Files
    http://schnaader.info
    Damn kids. They're all alike.

  35. Thanks:

    Krishty (28th November 2019)

  36. #56
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    93
    Thanks
    56
    Thanked 47 Times in 29 Posts
    Confirmed! Tausend Dank, schnaader!

    I think I need to start dissecting JPG files before optimizing them …

  37. #57
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    93
    Thanks
    56
    Thanked 47 Times in 29 Posts
    New version: https://papas-best.com/optimizer_en

    added SVG optimization via Leanify
    Not much to say here. Only supporting “real” SVG so far, no SVG embedded in XML & Co.

    removed TGA-to-PNG support (due to PNGOUT license)
    This never worked in the public build anyway, so I removed it.

    fixed crashes
    Stupid little synchronization issues. They crashed the Optimizer especially when optimizing many small files in rapid succession.

    fixed PNG optimization sometimes missing a 1-B opportunity
    The DeflOpt/defluff loop did not work properly when gains were below 1 B. On 250 files of mine, the new version improved compression on three. The gains were 1, 1, and 3 bytes. No big deal, but the improvement comes with virtually no cost.

    fixed very small window dimensions
    You can no longer resize the UI so small that it looks stupid.

  38. Thanks:

    necros (20th December 2019)

  39. #58
    Member
    Join Date
    Jul 2014
    Location
    Mars
    Posts
    206
    Thanks
    142
    Thanked 13 Times in 12 Posts
    Hm, RMB click on file llist area exits Papa`s, bug? (win10 x64)
    Feat. req.- Auto save settings

  40. Thanks:

    Krishty (20th December 2019)

  41. #59
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    93
    Thanks
    56
    Thanked 47 Times in 29 Posts
    Quote Originally Posted by necros View Post
    Hm, RMB click on file llist area exits Papa`s, bug? (win10 x64)Feat. req.- Auto save settings
    Definitely a bug. It doesn’t happen if you right-click a file name directly, only on the blank area (I hope!). Will be fixed, thanks!

  42. #60
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    93
    Thanks
    56
    Thanked 47 Times in 29 Posts
    necros, the bug is fixed in this version: https://papas-best.com/optimizer_en

    Thank you again for reporting it!

  43. Thanks:

    necros (22nd December 2019)

Page 2 of 4 FirstFirst 1234 LastLast

Similar Threads

  1. defluff - a deflate huffman optimizer
    By jo.henke in forum Data Compression
    Replies: 48
    Last Post: 7th November 2018, 01:04
  2. Does a jpeg2000 optimizer exist ?
    By SvenBent in forum The Off-Topic Lounge
    Replies: 2
    Last Post: 28th August 2014, 17:24
  3. Ultra7z Optimizer 0.12 is out
    By Surfer in forum Data Compression
    Replies: 13
    Last Post: 30th December 2010, 00:22
  4. Pak_Pic Gui for pictures optimizer
    By ben in forum Data Compression
    Replies: 0
    Last Post: 22nd November 2010, 20:36
  5. help : Psd optimizer
    By Yuri Grille. in forum Data Compression
    Replies: 25
    Last Post: 25th April 2009, 10:29

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •