Page 3 of 4 FirstFirst 1234 LastLast
Results 61 to 90 of 92

Thread: Papa’s Optimizer

  1. #61
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    93
    Thanks
    56
    Thanked 47 Times in 29 Posts
    Quote Originally Posted by necros View Post
    Feat. req.- Auto save settings
    Please check out the new version: https://papas-best.com/optimizer_en#download (or first post in this thread)

    UI now remembers settings on graceful exit
    The settings are stored in the registry under HKEY_CURRENT_USER/Software/Papa’s Best/Optimizer whenever you quit the program normally (i.e. if it doesn’t crash). This is experimental; please report any problems!

    fixed DMG settings being confused with RAR settings
    This was a typo in the code and it slipped through testing. Sorry!

    fixed create/modified dates not being preserved
    Depending on the specific toolset used, the file dates were not preserved. It seemed to work mostly for PNG, but many other files got the current date as modification/creation date. This regression was introduced with the new threading system, but dates should be preserved properly now.

  2. Thanks (3):

    JamesWasil (3rd January 2020),necros (3rd January 2020),zeon (3rd January 2020)

  3. #62
    Member CompressMaster's Avatar
    Join Date
    Jun 2018
    Location
    Lovinobana, Slovakia
    Posts
    216
    Thanks
    66
    Thanked 18 Times in 18 Posts
    Quote Originally Posted by Krishty View Post
    UI now remembers settings on graceful exit
    The settings are stored in the registry under HKEY_CURRENT_USER/Software/Papa’s Best/Optimizer whenever you quit the program normally (i.e. if it doesn’t crash). This is experimental; please report any problems!
    Well, I think I´d be better if settings will be stored in file.

  4. Thanks:

    necros (4th January 2020)

  5. #63
    Member
    Join Date
    Jun 2016
    Location
    Axis
    Posts
    7
    Thanks
    37
    Thanked 0 Times in 0 Posts
    I agree: an ini file or xml would be nice (next to the program's executable), as a matter of portability

  6. #64
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    93
    Thanks
    56
    Thanked 47 Times in 29 Posts
    Sorry but no, it will stay registry-based and I won’t bikeshed this.

    If you want it to be portable, consider that you can export and remove registry keys via REG files or command line with built-in Windows tools.

    (Had my share of file-based settings in software deployment / setup packaging. Will not do this ever again. At some point during software development, you just have to say no, please understand!)

  7. #65
    Member
    Join Date
    Jun 2016
    Location
    Axis
    Posts
    7
    Thanks
    37
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Krishty View Post
    Will not do this ever again. At some point during software development, you just have to say no, please understand!)
    Ok, but It's a pity: there are many programs (simple and complex) that are natively portable (and so useful on the USB drives inserted in various host pc): Blender, Total Commander, Audacity, foorbar2000, peazip, potplayer, irfanview, xnview, pdf-xchange viewer/editor, chromium browser, mirc, nirsoft's large numbers and many others.

    Anyway, ok, thanks, I don't insist

  8. #66
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    93
    Thanks
    56
    Thanked 47 Times in 29 Posts
    Yep, I love Nirsoft’s! Thanks. Let’s look forward to the interesting problems: MPO/APNG handling bugs, missing EPUB/PDF optimization

  9. #67
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    93
    Thanks
    56
    Thanked 47 Times in 29 Posts
    Prototype of EPUB optimization attached to this post. Optimizes the book’s ZIP container as well as contained PNG, JPG, SVG, XML, … seems to gain 5–10% on ordinary books.

    I can’t test the optimization reliably. There is a program called epubcheck, but I have some trouble setting it up. Will do this soon.

    The main problem with EPUB is the harsh restrictions on the ZIP container. The first file in the archive must be called mimetype and it must be uncompressed. Both requirements are hard to carry through the toolchain; for example, 7-Zip sorts files in ZIP archives by filename – no matter the command line, even breaking it into two steps doesn’t change this.

    I eventually settled with renaming mimetype to !!!!!!!!, a filename that will most likely be first. After all compression is done, I can use 7-Zip’s rename command to change the name back to mimetype – and this keeps the original file order. Then I overwrite the file in the archive via 7-Zip’s add, without compression. To my surprise, this also keeps the original order.

    On the command line, it’s something like
    Code:
    7z rn foo.zip !!!!!!!! mimetype
    7z a foo.zip mimetype -mm=Deflate -m0=Copy -mtc=off -mcl=off -mcu=off


    I’d be glad about feedback, considering especially that I can’t use the verification tool yet and my wife’s ebook reader is broken.
    Attached Files Attached Files

  10. Thanks:

    necros (13th December 2020)

  11. #68
    Member
    Join Date
    Jan 2020
    Location
    Fl
    Posts
    53
    Thanks
    1
    Thanked 3 Times in 3 Posts
    Hi Why does this program need to scan my files like my entire internet cache files before it does its job? I tried to test it to compress a small 200k jpeg file and had to stop it since it took over a minute.
    Also I use pnggauntlet for png images which consists of your optipng and others.
    thanks

  12. #69
    Member
    Join Date
    Jul 2014
    Location
    Mars
    Posts
    206
    Thanks
    142
    Thanked 13 Times in 12 Posts
    While optimizing renamed 17mb zip file with not zip extension (ZIP-like) after some time gave - status - exit code C0000005 , what does it mean?

  13. #70
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    93
    Thanks
    56
    Thanked 47 Times in 29 Posts
    Quote Originally Posted by Trench View Post
    Hi Why does this program need to scan my files like my entire internet cache files before it does its job? I tried to test it to compress a small 200k jpeg file and had to stop it since it took over a minute.
    Also I use pnggauntlet for png images which consists of your optipng and others.
    thanks
    Hi Trench!

    Sorry for that, it sounds like a bug – the scan should only start if you drag and drop a folder or if you use the Add Folder button. It shouldn’t scan anything for one file.

    Could you please describe what exactly you did? Did you drag & drop the JPEG to the Optimizer, or did you use Add File? If we can reproduce the problem, I’ll fix it.

    Quote Originally Posted by necros View Post
    While optimizing renamed 17mb zip file with not zip extension (ZIP-like) after some time gave - status - exit code C0000005 , what does it mean?
    Sounds like one of the Windows codes for Access Violations. Please right-click the affected file in the list, select Open log file, copy and paste it here (or send it via PM if there’s personal information in it).

  14. #71
    Member
    Join Date
    Jul 2014
    Location
    Mars
    Posts
    206
    Thanks
    142
    Thanked 13 Times in 12 Posts
    Leanify.exe --max_depth 1 "z:\TMP\papaA86BDFFD\x.zip"
    Processing: z:\TMP\papaA86BDFFD\x.zip

    exit code C0000005

    got it - leanify error - after manual optimization it finishes with no error but file is corrupted (

  15. #72
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    93
    Thanks
    56
    Thanked 47 Times in 29 Posts
    Quote Originally Posted by necros View Post
    Leanify.exe --max_depth 1 "z:\TMP\papaA86BDFFD\x.zip"
    Processing: z:\TMP\papaA86BDFFD\x.zip

    exit code C0000005

    got it - leanify error - after manual optimization it finishes with no error but file is corrupted (
    If you send me the file, I’ll check if the problem reproduces with the latest version and if so, I’ll report the error to JayXon.

    Just mentioning again that my optimizer uses a checksum to check validity of the optimized data, so even if Leanify corrupted the file silently your original would remain untouched

  16. #73
    Member
    Join Date
    Jan 2020
    Location
    Fl
    Posts
    53
    Thanks
    1
    Thanked 3 Times in 3 Posts
    Quote Originally Posted by Krishty View Post
    Sorry for that, it sounds like a bug – the scan should only start if you drag and drop a folder or if you use the Add Folder button. It shouldn’t scan anything for one file.

    Could you please describe what exactly you did? Did you drag & drop the JPEG to the Optimizer, or did you use Add File? If we can reproduce the problem, I’ll fix it..
    I dont remember how but maybe it was drag and drop or maybe I messed up.

    Also I assume in the future you have a clear all button to since i usually shut it off and load it again to see have it clear.
    I would be better if the instructions explained better what they do. Since one png file when I messed with the setting it increased the file size by over 900% it says from 300kb to 3mb. I think I pressed for 7zip and clear transparent.
    Many of the png files seem to get stuck if selected or dragged. While slows down the bulk process.
    it randomly gets stuck on jpeg images but when closing program and doing it again it works. drag and drop.
    for a small 10kb gif image (20 years old file) it took over 3 minute to get to 30%. which does not do well with gif.
    While this thing takes forever on small animated gif compression and doc files.
    It does a good job converting other gif files to half sometimes.
    hope that helps

  17. #74
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    93
    Thanks
    56
    Thanked 47 Times in 29 Posts
    Yeah, absolutely. I guess I’ll add tooltips or help to the settings in future versions. However, every bit of UI work keeps me from implementing actual optimization.

    You can fix the increased PNGs by 1) disabling PNG → for 7z 2) enabling General → Force (even if already optimized) and 3) optimizing the affected files again.

    When you mention it gets stuck on JPEG randomly, does the whole program get stuck or only the JPEG optimization? There is a problem with ExifTool that causes random failures on JPEG, often with "exit code 0x000000FF" messages. I’m desperately trying to work around this using ECT because ExifTool is a huge train wreck, but it’s difficult.

    DOC and ZIP will be substantially faster in a future version. There was some confusion over ECT optimizations that caused this. GIF is very aggressive and takes very long on animated images, yes. So does PNG. It doesn’t get stuck – and actually ECT just got a small improvement so it looks less frozen in my Optimizer – it optimizes down to the very last bytes. If you want less compression and more speed, use another tool, like Nikkho’s Optimizer.

  18. #75
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    93
    Thanks
    56
    Thanked 47 Times in 29 Posts
    New version: https://papas-best.com/optimizer_en#download

    added EPUB optimization

    I tested the optimization on 210 ebooks. Optimization took ~5 hours with 10 threads on a Ryzen 3600. Size shrank from 135 MiB to 121 MiB, or 90.1%.

    I ran the original ebooks as well as the optimized ones through epubcheck. The output before and after was identical, so my optimizations seem legit. I didn’t get any complaints over the beta version I uploaded either.

    Individual compression steps FYI:
    • Contained JPG is optimized. All metadata is removed (useless for EPUB), except for rotation info (since EPUB is browser-based, the layout likely builds upon rotation metadata just like normal browsers do).
    • Contained PNG is optimized. All metadata is removed as is colors of fully-transparent pixels.
    • Contained GIF is optimized. If not animated, it is converted to PNG. Both formats are part of the EPUB specification and readers don’t care as long as filenames and dimensions match.
    • Contained XML/HTML/SVG is minified via Leanify.
    • All other contained filetypes are optimized in fully lossless mode (i.e. no BMP-to-PNG conversion, ZIP timestamp removal or similar lossy tricks). The reason being, EPUB allows such non-standard types for supplemental purposes like attaching data tables to scientific papers; I won’t break what I don’t understand.
    • The ZIP container is optimized according to what I explained earler.


    fixed dead worker threads if errors occurred during archive handling

    When processing damaged archives (ZIP, gzip, Office stuff), worker threads would fall asleep and never wake up again. This was a stupid mistake in error handling and is now fixed.

  19. Thanks:

    necros (2nd February 2020)

  20. #76
    Member Jaff's Avatar
    Join Date
    Oct 2012
    Location
    Dracula's country
    Posts
    104
    Thanks
    116
    Thanked 22 Times in 18 Posts
    1) Can optimised JPEG thumbnails be trimmed to needed size and cut the zero paddings? Thank you!
    2) Option to keep JPEG trailings? (Samsung uses sometimes to store makernotes (example: J320F), interactive panorama MP4 movie (example: Galaxy S7); and aditional info in Samsung trailer format: accurate time in epoch format, panoramas info (all mobile camera photos in panorama mode), Samsung Gear 360 trailer, etc)
    Last edited by Jaff; 6th February 2020 at 12:24.

  21. #77
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    93
    Thanks
    56
    Thanked 47 Times in 29 Posts
    Quote Originally Posted by Jaff View Post
    1) Can optimised JPEG thumbnails be trimmed to needed size and cut the zero paddings? Thank you!
    I currently only pass those things to ECT, so this is a huge task.
    Quote Originally Posted by Jaff View Post
    2) Option to keep JPEG trailings? (Samsung uses sometimes to store makernotes (example: J320F), interactive panorama MP4 movie (example: Galaxy S7); and aditional info in Samsung trailer format: accurate time in epoch format, panoramas info (all mobile camera photos in panorama mode), Samsung Gear 360 trailer, etc)
    Yeah, I encountered this myself: https://encode.su/threads/3186-Papa%...ll=1#post62394

    But now I need to start parsing JPG myself before handing it over to ECT etc., so … expect it to take quite some time.

  22. #78
    Member Jaff's Avatar
    Join Date
    Oct 2012
    Location
    Dracula's country
    Posts
    104
    Thanks
    116
    Thanked 22 Times in 18 Posts
    Quote Originally Posted by Krishty View Post
    I currently only pass those things to ECT, so this is a huge task.
    No problem. I found a way to fix thumbnail optimization using exiftool!
    1) optimize JPEG
    2) extract thumbnail into a secondary file ( exiftool -b -ThumbnailImage %1>%2 ) %1 = JPEG, %2 = thumbnail file
    3) optimize thumbnail image (to cut zero padding)
    4) put back thumbnail ( exiftool "-ThumbnailImage<=%1" %2 ) %1 = thumbnail file, %2 = JPEG

    Done!

    edit: for keeping JPEG trailers it will be much easier with the new version of exiftool - https://exiftool.org/forum/index.php?topic=10813.0
    Last edited by Jaff; 8th February 2020 at 04:05.

  23. Thanks:

    Krishty (8th February 2020)

  24. #79
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    93
    Thanks
    56
    Thanked 47 Times in 29 Posts
    Amazing hint, Jaff, thanks! To move the thumbnail back, you’ll have to put the "-ThumbnailImage<=%1" part into actual quotes.

    I’ll try to quickly build that in.

  25. Thanks:

    Jaff (8th February 2020)

  26. #80
    Member Jaff's Avatar
    Join Date
    Oct 2012
    Location
    Dracula's country
    Posts
    104
    Thanks
    116
    Thanked 22 Times in 18 Posts
    Put this before optimize JPEG:
    0) Extract JPEG trailer to separate file (need latest exiftool 11.87)
    exiftool -b -trailer 1.jpg >2.dat
    5) add back the trailer:
    copy /b 1.jpg +2.dat 3.jpg
    1.jpg = optimised file, 2.dat (saved trailer); 3.jpg (new optimized file)
    Let's rename file back and delete temp files...
    del 1.jpg
    del 2.dat
    ren 3.jpg 1.jpg
    I wait for the new version. Now you can add option to strip the trailers or not.

  27. Thanks:

    Krishty (18th February 2020)

  28. #81
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    93
    Thanks
    56
    Thanked 47 Times in 29 Posts
    I did some experiments with huffmix according to this post by SvenBent: https://encode.su/threads/2274-ECT-a...ll=1#post64959

    (There is no public build because I haven’t gotten a response from caveman so far regarding the huffmix license.)

    I tested a few thousand PNGs from my hard drive. Previous optimization used ECT + defluff + DeflOpt; now it uses huffmix on all intermediate results. Some observations:

    • Without pngout, huffmix has only three streams to choose from: ECT output, ECT + DeflOpt, ECT + defluff + DeflOpt. So there is not much gain to expect.

    • Actual gains were seen in about one out of fifty files. These were mostly 1-B gains; one file got 7 B smaller and another 13 B. The larger the file, the larger the gains.

    • The error rate increased significantly:
      • “More than 1024 Deflate blocks detected, this is not handled by this version.” (known error with large PNGs)
      • “Type 0 (uncompressed) block detected, this is not handled by this version.” (known error)
      • On a few files, huffmix terminated without any error message


    • There is a huge increase in complexity:
      • Previously, there was just one pipeline for all DEFLATE-based formats. I have to maintain a separate path for ZIP now, which is supported by ECT/defluff/DeflOpt but not by huffmix.
      • If huffmix exits with error code 0, all is well.
      • Else if huffmix exits with error code 1, parse stdout:
        • If stdout contains “Type 0 (uncompressed) block detected, this is not handled by this version.”, we have a soft error. Just pick the smaller file.
        • Else if stdout contains “More than 1024 Deflate blocks detected, this is not handled by this version.”, we have a soft error. Just pick the smaller file.
        • Else if stderr (not stdout!) contains “is an unknown file type”, we have hit a file that huffmix doesn’t understand. Just pick the smaller file. (This also happens with defluff and archives containing some Unicode characters.)
        • Else we have a hard error like a destroyed file; abort.


      • There is much more file I/O going on, and there seems to be a very rare race condition with Win32’s CopyFile and new processes. Not huffmix’s fault, but something that is now being triggered much more often.

    All in all, I’m not sure I should keep working on it. It definitely is a great tool, but it comes with so many limitations and rough edges that the few bytes it gains me over my existing pipeline hardly justify the increased complexity.

  29. #82
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    926
    Thanks
    58
    Thanked 116 Times in 93 Posts
    Quote Originally Posted by Krishty View Post
    I[*]The error rate increased significantly:
    • “More than 1024 Deflate blocks detected, this is not handled by this version.” (known error with large PNGs)
    • “Type 0 (uncompressed) block detected, this is not handled by this version.” (known error)
    • On a few files, huffmix terminated without any error message
    I can confirm both issues with huffmix and ECT on "big" files
    apparently ECT just kills its with number of deflate blocks.
    it was never an issue for me on pngout

    I've notified the developer in his forum post about it

    What I was really curious about is to do the Final ECT mixed with pngout as long as filters and palettes stay the same i guess it could be possible. but I haven't tested it much and success rate has been mixed due to the the 2 issues above

  30. #83
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    93
    Thanks
    56
    Thanked 47 Times in 29 Posts
    Quote Originally Posted by Krishty View Post
    I’ll try to quickly build that in.
    … and then came the COVID lockdown …

    I haven’t forgotten you, Jaff. I haven’t implemented the trailing data handling yet, but your thumbnail optimization suggestion works really well. Find attached the portable version of the current build; the setup/update can be found on my site.

    Feedback is always welcome. I notice that JPEG optimization is somewhat slower now, because ExifTool (which is my bottleneck) runs once more per file.

    Changes:
    • added JPEG thumbnail optimization
    • added option to remove Adobe metadata from JPEG
      This was suggested to me and it indeed strips some files further. ExifTool doesn’t do it by default. Be aware, though, that there is a good reason for that: The data may be required to correctly identify the image’s color space. If you use this option, be sure to check the result; especially if you use CMYK & Co a lot. For an example of what could go wrong, see here.
    • improved launch speed
    • improved performance during resizing
    • updated ECT to 0.8.3
    • updated ExifTool to 12.03
    • fixed genetic PNG optimization appearing frozen
      Thanks to fhanau for the help!
    • fixed wrong status messages
    • minor performance improvements
    Attached Files Attached Files

  31. Thanks (4):

    dado023 (4th August 2020),Jaff (15th January 2021),Mike (4th August 2020),necros (13th December 2020)

  32. #84
    Member
    Join Date
    Mar 2016
    Location
    Croatia
    Posts
    195
    Thanks
    82
    Thanked 14 Times in 12 Posts
    Does this compresses higher than Pingo ?

  33. #85
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    93
    Thanks
    56
    Thanked 47 Times in 29 Posts
    Quote Originally Posted by dado023 View Post
    Does this compresses higher than Pingo ?
    I don’t know If you or someone else would like to go ahead and benchmark it, I’d be very happy to learn! I’m using ECT and I’ve seen Pingo talk in the ECT thread, but to be honest I didn’t understand the bottom line.

  34. #86
    Member
    Join Date
    Mar 2016
    Location
    Croatia
    Posts
    195
    Thanks
    82
    Thanked 14 Times in 12 Posts
    Hm.....do we(ppl on forum) have standardized test collection of PNGs...i am willing to download and test, just provide me with samples to test with.

  35. #87
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    93
    Thanks
    56
    Thanked 47 Times in 29 Posts
    Minor update and hotfix:

    • ​fixed some TIFFs being destroyed during optimization
      OptiPNG’s -fix option normally helps me recover BMP/GIF/TIFF files with minor damage, but I have found that it destroys some images silently. One such image is attached.
      This may happen inside of Office documents, where the corruption is hard to find for the user. Therefore, I disabled this feature until I find a more reliable way to do it. Optimizer will now fail on such files.
    • added a button to change the temp directory
      The temp directory feature never worked quite right; it is now fixed and easier to use.
    • fixed temp directory not being saved correctly
    • fixed Adobe metadata option not being saved correctly
    • minor performance improvements
    • minor setup improvements
      Slightly improved compression ratio on the 7-Zip download and the MSI. Small changes to the way the MSI stores paths to avoid redundancy.

    The new version can, as always, be downloaded from https://papas-best.com/optimizer_en.

    I’ll try to add it to the first post here as well, but this always breaks formatting, so we’ll see how it goes.
    Attached Files Attached Files
    Last edited by Krishty; 8th December 2020 at 02:56.

  36. Thanks:

    necros (13th December 2020)

  37. #88
    Member
    Join Date
    Jul 2014
    Location
    Mars
    Posts
    206
    Thanks
    142
    Thanked 13 Times in 12 Posts
    Can new version email notification possibly be implemented in near future?

  38. #89
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    93
    Thanks
    56
    Thanked 47 Times in 29 Posts
    Quote Originally Posted by necros View Post
    Can new version email notification possibly be implemented in near future?
    Good idea, that never occurred to me!

    Would an RSS feed be sufficient? Mail clients like Thunderbird or Windows Mail App support RSS just like mail, and I wouldn’t have to store email addresses (better privacy).

    I already maintain an RSS feed under https://papas-best.com/feed , but I could add another just for Optimizer changelogs – one notification per new version, with changes and download link.

  39. Thanks:

    Mike (14th December 2020)

  40. #90
    Member
    Join Date
    Jul 2014
    Location
    Mars
    Posts
    206
    Thanks
    142
    Thanked 13 Times in 12 Posts
    We have to try it, can your recommend some good RSS client for Win?

Page 3 of 4 FirstFirst 1234 LastLast

Similar Threads

  1. defluff - a deflate huffman optimizer
    By jo.henke in forum Data Compression
    Replies: 48
    Last Post: 7th November 2018, 01:04
  2. Does a jpeg2000 optimizer exist ?
    By SvenBent in forum The Off-Topic Lounge
    Replies: 2
    Last Post: 28th August 2014, 17:24
  3. Ultra7z Optimizer 0.12 is out
    By Surfer in forum Data Compression
    Replies: 13
    Last Post: 30th December 2010, 00:22
  4. Pak_Pic Gui for pictures optimizer
    By ben in forum Data Compression
    Replies: 0
    Last Post: 22nd November 2010, 20:36
  5. help : Psd optimizer
    By Yuri Grille. in forum Data Compression
    Replies: 25
    Last Post: 25th April 2009, 10:29

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •