Page 3 of 3 FirstFirst 123
Results 61 to 82 of 82

Thread: Papa’s Optimizer

  1. #61
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    86
    Thanks
    55
    Thanked 39 Times in 24 Posts
    Quote Originally Posted by necros View Post
    Feat. req.- Auto save settings
    Please check out the new version: https://papas-best.com/optimizer_en#download (or first post in this thread)

    UI now remembers settings on graceful exit
    The settings are stored in the registry under HKEY_CURRENT_USER/Software/Papa’s Best/Optimizer whenever you quit the program normally (i.e. if it doesn’t crash). This is experimental; please report any problems!

    fixed DMG settings being confused with RAR settings
    This was a typo in the code and it slipped through testing. Sorry!

    fixed create/modified dates not being preserved
    Depending on the specific toolset used, the file dates were not preserved. It seemed to work mostly for PNG, but many other files got the current date as modification/creation date. This regression was introduced with the new threading system, but dates should be preserved properly now.

  2. Thanks (3):

    JamesWasil (3rd January 2020),necros (3rd January 2020),zeon (3rd January 2020)

  3. #62
    Member CompressMaster's Avatar
    Join Date
    Jun 2018
    Location
    Lovinobana, Slovakia
    Posts
    189
    Thanks
    52
    Thanked 13 Times in 13 Posts
    Quote Originally Posted by Krishty View Post
    UI now remembers settings on graceful exit
    The settings are stored in the registry under HKEY_CURRENT_USER/Software/Papa’s Best/Optimizer whenever you quit the program normally (i.e. if it doesn’t crash). This is experimental; please report any problems!
    Well, I think I´d be better if settings will be stored in file.

  4. Thanks:

    necros (4th January 2020)

  5. #63
    Member
    Join Date
    Jun 2016
    Location
    Axis
    Posts
    7
    Thanks
    37
    Thanked 0 Times in 0 Posts
    I agree: an ini file or xml would be nice (next to the program's executable), as a matter of portability

  6. #64
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    86
    Thanks
    55
    Thanked 39 Times in 24 Posts
    Sorry but no, it will stay registry-based and I won’t bikeshed this.

    If you want it to be portable, consider that you can export and remove registry keys via REG files or command line with built-in Windows tools.

    (Had my share of file-based settings in software deployment / setup packaging. Will not do this ever again. At some point during software development, you just have to say no, please understand!)

  7. #65
    Member
    Join Date
    Jun 2016
    Location
    Axis
    Posts
    7
    Thanks
    37
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Krishty View Post
    Will not do this ever again. At some point during software development, you just have to say no, please understand!)
    Ok, but It's a pity: there are many programs (simple and complex) that are natively portable (and so useful on the USB drives inserted in various host pc): Blender, Total Commander, Audacity, foorbar2000, peazip, potplayer, irfanview, xnview, pdf-xchange viewer/editor, chromium browser, mirc, nirsoft's large numbers and many others.

    Anyway, ok, thanks, I don't insist

  8. #66
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    86
    Thanks
    55
    Thanked 39 Times in 24 Posts
    Yep, I love Nirsoft’s! Thanks. Let’s look forward to the interesting problems: MPO/APNG handling bugs, missing EPUB/PDF optimization

  9. #67
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    86
    Thanks
    55
    Thanked 39 Times in 24 Posts
    Prototype of EPUB optimization attached to this post. Optimizes the book’s ZIP container as well as contained PNG, JPG, SVG, XML, … seems to gain 5–10% on ordinary books.

    I can’t test the optimization reliably. There is a program called epubcheck, but I have some trouble setting it up. Will do this soon.

    The main problem with EPUB is the harsh restrictions on the ZIP container. The first file in the archive must be called mimetype and it must be uncompressed. Both requirements are hard to carry through the toolchain; for example, 7-Zip sorts files in ZIP archives by filename – no matter the command line, even breaking it into two steps doesn’t change this.

    I eventually settled with renaming mimetype to !!!!!!!!, a filename that will most likely be first. After all compression is done, I can use 7-Zip’s rename command to change the name back to mimetype – and this keeps the original file order. Then I overwrite the file in the archive via 7-Zip’s add, without compression. To my surprise, this also keeps the original order.

    On the command line, it’s something like
    Code:
    7z rn foo.zip !!!!!!!! mimetype
    7z a foo.zip mimetype -mm=Deflate -m0=Copy -mtc=off -mcl=off -mcu=off


    I’d be glad about feedback, considering especially that I can’t use the verification tool yet and my wife’s ebook reader is broken.
    Attached Files Attached Files

  10. #68
    Member
    Join Date
    Jan 2020
    Location
    Fl
    Posts
    32
    Thanks
    0
    Thanked 2 Times in 2 Posts
    Hi Why does this program need to scan my files like my entire internet cache files before it does its job? I tried to test it to compress a small 200k jpeg file and had to stop it since it took over a minute.
    Also I use pnggauntlet for png images which consists of your optipng and others.
    thanks

  11. #69
    Member
    Join Date
    Jul 2014
    Location
    Mars
    Posts
    195
    Thanks
    133
    Thanked 13 Times in 12 Posts
    While optimizing renamed 17mb zip file with not zip extension (ZIP-like) after some time gave - status - exit code C0000005 , what does it mean?

  12. #70
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    86
    Thanks
    55
    Thanked 39 Times in 24 Posts
    Quote Originally Posted by Trench View Post
    Hi Why does this program need to scan my files like my entire internet cache files before it does its job? I tried to test it to compress a small 200k jpeg file and had to stop it since it took over a minute.
    Also I use pnggauntlet for png images which consists of your optipng and others.
    thanks
    Hi Trench!

    Sorry for that, it sounds like a bug – the scan should only start if you drag and drop a folder or if you use the Add Folder button. It shouldn’t scan anything for one file.

    Could you please describe what exactly you did? Did you drag & drop the JPEG to the Optimizer, or did you use Add File? If we can reproduce the problem, I’ll fix it.

    Quote Originally Posted by necros View Post
    While optimizing renamed 17mb zip file with not zip extension (ZIP-like) after some time gave - status - exit code C0000005 , what does it mean?
    Sounds like one of the Windows codes for Access Violations. Please right-click the affected file in the list, select Open log file, copy and paste it here (or send it via PM if there’s personal information in it).

  13. #71
    Member
    Join Date
    Jul 2014
    Location
    Mars
    Posts
    195
    Thanks
    133
    Thanked 13 Times in 12 Posts
    Leanify.exe --max_depth 1 "z:\TMP\papaA86BDFFD\x.zip"
    Processing: z:\TMP\papaA86BDFFD\x.zip

    exit code C0000005

    got it - leanify error - after manual optimization it finishes with no error but file is corrupted (

  14. #72
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    86
    Thanks
    55
    Thanked 39 Times in 24 Posts
    Quote Originally Posted by necros View Post
    Leanify.exe --max_depth 1 "z:\TMP\papaA86BDFFD\x.zip"
    Processing: z:\TMP\papaA86BDFFD\x.zip

    exit code C0000005

    got it - leanify error - after manual optimization it finishes with no error but file is corrupted (
    If you send me the file, I’ll check if the problem reproduces with the latest version and if so, I’ll report the error to JayXon.

    Just mentioning again that my optimizer uses a checksum to check validity of the optimized data, so even if Leanify corrupted the file silently your original would remain untouched

  15. #73
    Member
    Join Date
    Jan 2020
    Location
    Fl
    Posts
    32
    Thanks
    0
    Thanked 2 Times in 2 Posts
    Quote Originally Posted by Krishty View Post
    Sorry for that, it sounds like a bug – the scan should only start if you drag and drop a folder or if you use the Add Folder button. It shouldn’t scan anything for one file.

    Could you please describe what exactly you did? Did you drag & drop the JPEG to the Optimizer, or did you use Add File? If we can reproduce the problem, I’ll fix it..
    I dont remember how but maybe it was drag and drop or maybe I messed up.

    Also I assume in the future you have a clear all button to since i usually shut it off and load it again to see have it clear.
    I would be better if the instructions explained better what they do. Since one png file when I messed with the setting it increased the file size by over 900% it says from 300kb to 3mb. I think I pressed for 7zip and clear transparent.
    Many of the png files seem to get stuck if selected or dragged. While slows down the bulk process.
    it randomly gets stuck on jpeg images but when closing program and doing it again it works. drag and drop.
    for a small 10kb gif image (20 years old file) it took over 3 minute to get to 30%. which does not do well with gif.
    While this thing takes forever on small animated gif compression and doc files.
    It does a good job converting other gif files to half sometimes.
    hope that helps

  16. #74
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    86
    Thanks
    55
    Thanked 39 Times in 24 Posts
    Yeah, absolutely. I guess I’ll add tooltips or help to the settings in future versions. However, every bit of UI work keeps me from implementing actual optimization.

    You can fix the increased PNGs by 1) disabling PNG → for 7z 2) enabling General → Force (even if already optimized) and 3) optimizing the affected files again.

    When you mention it gets stuck on JPEG randomly, does the whole program get stuck or only the JPEG optimization? There is a problem with ExifTool that causes random failures on JPEG, often with "exit code 0x000000FF" messages. I’m desperately trying to work around this using ECT because ExifTool is a huge train wreck, but it’s difficult.

    DOC and ZIP will be substantially faster in a future version. There was some confusion over ECT optimizations that caused this. GIF is very aggressive and takes very long on animated images, yes. So does PNG. It doesn’t get stuck – and actually ECT just got a small improvement so it looks less frozen in my Optimizer – it optimizes down to the very last bytes. If you want less compression and more speed, use another tool, like Nikkho’s Optimizer.

  17. #75
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    86
    Thanks
    55
    Thanked 39 Times in 24 Posts
    New version: https://papas-best.com/optimizer_en#download

    added EPUB optimization

    I tested the optimization on 210 ebooks. Optimization took ~5 hours with 10 threads on a Ryzen 3600. Size shrank from 135 MiB to 121 MiB, or 90.1%.

    I ran the original ebooks as well as the optimized ones through epubcheck. The output before and after was identical, so my optimizations seem legit. I didn’t get any complaints over the beta version I uploaded either.

    Individual compression steps FYI:
    • Contained JPG is optimized. All metadata is removed (useless for EPUB), except for rotation info (since EPUB is browser-based, the layout likely builds upon rotation metadata just like normal browsers do).
    • Contained PNG is optimized. All metadata is removed as is colors of fully-transparent pixels.
    • Contained GIF is optimized. If not animated, it is converted to PNG. Both formats are part of the EPUB specification and readers don’t care as long as filenames and dimensions match.
    • Contained XML/HTML/SVG is minified via Leanify.
    • All other contained filetypes are optimized in fully lossless mode (i.e. no BMP-to-PNG conversion, ZIP timestamp removal or similar lossy tricks). The reason being, EPUB allows such non-standard types for supplemental purposes like attaching data tables to scientific papers; I won’t break what I don’t understand.
    • The ZIP container is optimized according to what I explained earler.


    fixed dead worker threads if errors occurred during archive handling

    When processing damaged archives (ZIP, gzip, Office stuff), worker threads would fall asleep and never wake up again. This was a stupid mistake in error handling and is now fixed.

  18. Thanks:

    necros (2nd February 2020)

  19. #76
    Member Jaff's Avatar
    Join Date
    Oct 2012
    Location
    Dracula's country
    Posts
    104
    Thanks
    115
    Thanked 22 Times in 18 Posts
    1) Can optimised JPEG thumbnails be trimmed to needed size and cut the zero paddings? Thank you!
    2) Option to keep JPEG trailings? (Samsung uses sometimes to store makernotes (example: J320F), interactive panorama MP4 movie (example: Galaxy S7); and aditional info in Samsung trailer format: accurate time in epoch format, panoramas info (all mobile camera photos in panorama mode), Samsung Gear 360 trailer, etc)
    Last edited by Jaff; 6th February 2020 at 11:24.

  20. #77
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    86
    Thanks
    55
    Thanked 39 Times in 24 Posts
    Quote Originally Posted by Jaff View Post
    1) Can optimised JPEG thumbnails be trimmed to needed size and cut the zero paddings? Thank you!
    I currently only pass those things to ECT, so this is a huge task.
    Quote Originally Posted by Jaff View Post
    2) Option to keep JPEG trailings? (Samsung uses sometimes to store makernotes (example: J320F), interactive panorama MP4 movie (example: Galaxy S7); and aditional info in Samsung trailer format: accurate time in epoch format, panoramas info (all mobile camera photos in panorama mode), Samsung Gear 360 trailer, etc)
    Yeah, I encountered this myself: https://encode.su/threads/3186-Papa%...ll=1#post62394

    But now I need to start parsing JPG myself before handing it over to ECT etc., so … expect it to take quite some time.

  21. #78
    Member Jaff's Avatar
    Join Date
    Oct 2012
    Location
    Dracula's country
    Posts
    104
    Thanks
    115
    Thanked 22 Times in 18 Posts
    Quote Originally Posted by Krishty View Post
    I currently only pass those things to ECT, so this is a huge task.
    No problem. I found a way to fix thumbnail optimization using exiftool!
    1) optimize JPEG
    2) extract thumbnail into a secondary file ( exiftool -b -ThumbnailImage %1>%2 ) %1 = JPEG, %2 = thumbnail file
    3) optimize thumbnail image (to cut zero padding)
    4) put back thumbnail ( exiftool "-ThumbnailImage<=%1" %2 ) %1 = thumbnail file, %2 = JPEG

    Done!

    edit: for keeping JPEG trailers it will be much easier with the new version of exiftool - https://exiftool.org/forum/index.php?topic=10813.0
    Last edited by Jaff; 8th February 2020 at 03:05.

  22. Thanks:

    Krishty (8th February 2020)

  23. #79
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    86
    Thanks
    55
    Thanked 39 Times in 24 Posts
    Amazing hint, Jaff, thanks! To move the thumbnail back, you’ll have to put the "-ThumbnailImage<=%1" part into actual quotes.

    I’ll try to quickly build that in.

  24. Thanks:

    Jaff (8th February 2020)

  25. #80
    Member Jaff's Avatar
    Join Date
    Oct 2012
    Location
    Dracula's country
    Posts
    104
    Thanks
    115
    Thanked 22 Times in 18 Posts
    Put this before optimize JPEG:
    0) Extract JPEG trailer to separate file (need latest exiftool 11.87)
    exiftool -b -trailer 1.jpg >2.dat
    5) add back the trailer:
    copy /b 1.jpg +2.dat 3.jpg
    1.jpg = optimised file, 2.dat (saved trailer); 3.jpg (new optimized file)
    Let's rename file back and delete temp files...
    del 1.jpg
    del 2.dat
    ren 3.jpg 1.jpg
    I wait for the new version. Now you can add option to strip the trailers or not.

  26. Thanks:

    Krishty (18th February 2020)

  27. #81
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    86
    Thanks
    55
    Thanked 39 Times in 24 Posts
    I did some experiments with huffmix according to this post by SvenBent: https://encode.su/threads/2274-ECT-a...ll=1#post64959

    (There is no public build because I haven’t gotten a response from caveman so far regarding the huffmix license.)

    I tested a few thousand PNGs from my hard drive. Previous optimization used ECT + defluff + DeflOpt; now it uses huffmix on all intermediate results. Some observations:

    • Without pngout, huffmix has only three streams to choose from: ECT output, ECT + DeflOpt, ECT + defluff + DeflOpt. So there is not much gain to expect.

    • Actual gains were seen in about one out of fifty files. These were mostly 1-B gains; one file got 7 B smaller and another 13 B. The larger the file, the larger the gains.

    • The error rate increased significantly:
      • “More than 1024 Deflate blocks detected, this is not handled by this version.” (known error with large PNGs)
      • “Type 0 (uncompressed) block detected, this is not handled by this version.” (known error)
      • On a few files, huffmix terminated without any error message


    • There is a huge increase in complexity:
      • Previously, there was just one pipeline for all DEFLATE-based formats. I have to maintain a separate path for ZIP now, which is supported by ECT/defluff/DeflOpt but not by huffmix.
      • If huffmix exits with error code 0, all is well.
      • Else if huffmix exits with error code 1, parse stdout:
        • If stdout contains “Type 0 (uncompressed) block detected, this is not handled by this version.”, we have a soft error. Just pick the smaller file.
        • Else if stdout contains “More than 1024 Deflate blocks detected, this is not handled by this version.”, we have a soft error. Just pick the smaller file.
        • Else if stderr (not stdout!) contains “is an unknown file type”, we have hit a file that huffmix doesn’t understand. Just pick the smaller file. (This also happens with defluff and archives containing some Unicode characters.)
        • Else we have a hard error like a destroyed file; abort.


      • There is much more file I/O going on, and there seems to be a very rare race condition with Win32’s CopyFile and new processes. Not huffmix’s fault, but something that is now being triggered much more often.

    All in all, I’m not sure I should keep working on it. It definitely is a great tool, but it comes with so many limitations and rough edges that the few bytes it gains me over my existing pipeline hardly justify the increased complexity.

  28. #82
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    919
    Thanks
    57
    Thanked 113 Times in 90 Posts
    Quote Originally Posted by Krishty View Post
    I[*]The error rate increased significantly:
    • “More than 1024 Deflate blocks detected, this is not handled by this version.” (known error with large PNGs)
    • “Type 0 (uncompressed) block detected, this is not handled by this version.” (known error)
    • On a few files, huffmix terminated without any error message
    I can confirm both issues with huffmix and ECT on "big" files
    apparently ECT just kills its with number of deflate blocks.
    it was never an issue for me on pngout

    I've notified the developer in his forum post about it

    What I was really curious about is to do the Final ECT mixed with pngout as long as filters and palettes stay the same i guess it could be possible. but I haven't tested it much and success rate has been mixed due to the the 2 issues above

Page 3 of 3 FirstFirst 123

Similar Threads

  1. defluff - a deflate huffman optimizer
    By jo.henke in forum Data Compression
    Replies: 48
    Last Post: 7th November 2018, 00:04
  2. Does a jpeg2000 optimizer exist ?
    By SvenBent in forum The Off-Topic Lounge
    Replies: 2
    Last Post: 28th August 2014, 16:24
  3. Ultra7z Optimizer 0.12 is out
    By Surfer in forum Data Compression
    Replies: 13
    Last Post: 29th December 2010, 23:22
  4. Pak_Pic Gui for pictures optimizer
    By ben in forum Data Compression
    Replies: 0
    Last Post: 22nd November 2010, 19:36
  5. help : Psd optimizer
    By Yuri Grille. in forum Data Compression
    Replies: 25
    Last Post: 25th April 2009, 09:29

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •