Papa’s Optimizer

[Krishty]

Feat. req.- Auto save settings

Please check out the new version: https://papas-best.com/optimizer_en#download (or first post in this thread)

UI now remembers settings on graceful exit
The settings are stored in the registry under HKEY_CURRENT_USER/Software/Papa’s Best/Optimizer whenever you quit the program normally (i.e. if it doesn’t crash). This is experimental; please report any problems!

fixed DMG settings being confused with RAR settings
This was a typo in the code and it slipped through testing. Sorry!

fixed create/modified dates not being preserved
Depending on the specific toolset used, the file dates were not preserved. It seemed to work mostly for PNG, but many other files got the current date as modification/creation date. This regression was introduced with the new threading system, but dates should be preserved properly now.

[CompressMaster]

UI now remembers settings on graceful exit
The settings are stored in the registry under HKEY_CURRENT_USER/Software/Papa’s Best/Optimizer whenever you quit the program normally (i.e. if it doesn’t crash). This is experimental; please report any problems!

Well, I think I´d be better if settings will be stored in file.

[zeon]

I agree: an ini file or xml would be nice (next to the program's executable), as a matter of portability

[Krishty]

Sorry but no, it will stay registry-based and I won’t bikeshed this.

If you want it to be portable, consider that you can export and remove registry keys via REG files or command line with built-in Windows tools.

(Had my share of file-based settings in software deployment / setup packaging. Will not do this ever again. At some point during software development, you just have to say no, please understand!)

[zeon]

Will not do this ever again. At some point during software development, you just have to say no, please understand!)

Ok, but It's a pity: there are many programs (simple and complex) that are natively portable (and so useful on the USB drives inserted in various host pc): Blender, Total Commander, Audacity, foorbar2000, peazip, potplayer, irfanview, xnview, pdf-xchange viewer/editor, chromium browser, mirc, nirsoft's large numbers and many others.

Anyway, ok, thanks, I don't insist

[Krishty]

Yep, I love Nirsoft’s! Thanks. Let’s look forward to the interesting problems: MPO/APNG handling bugs, missing EPUB/PDF optimization

[Krishty]

Prototype of EPUB optimization attached to this post. Optimizes the book’s ZIP container as well as contained PNG, JPG, SVG, XML, … seems to gain 5–10% on ordinary books.

I can’t test the optimization reliably. There is a program called epubcheck, but I have some trouble setting it up. Will do this soon.

The main problem with EPUB is the harsh restrictions on the ZIP container. The first file in the archive must be called mimetype and it must be uncompressed. Both requirements are hard to carry through the toolchain; for example, 7-Zip sorts files in ZIP archives by filename – no matter the command line, even breaking it into two steps doesn’t change this.

I eventually settled with renaming mimetype to !!!!!!!!, a filename that will most likely be first. After all compression is done, I can use 7-Zip’s rename command to change the name back to mimetype – and this keeps the original file order. Then I overwrite the file in the archive via 7-Zip’s add, without compression. To my surprise, this also keeps the original order.

On the command line, it’s something like

7z rn foo.zip !!!!!!!! mimetype
7z a foo.zip mimetype -mm=Deflate -m0=Copy -mtc=off -mcl=off -mcu=off

I’d be glad about feedback, considering especially that I can’t use the verification tool yet and my wife’s ebook reader is broken.

[Trench]

Hi Why does this program need to scan my files like my entire internet cache files before it does its job? I tried to test it to compress a small 200k jpeg file and had to stop it since it took over a minute.
Also I use pnggauntlet for png images which consists of your optipng and others.
thanks

[necros]

While optimizing renamed 17mb zip file with not zip extension (ZIP-like) after some time gave - status - exit code C0000005 , what does it mean?

[Krishty]

Hi Why does this program need to scan my files like my entire internet cache files before it does its job? I tried to test it to compress a small 200k jpeg file and had to stop it since it took over a minute.
Also I use pnggauntlet for png images which consists of your optipng and others.
thanks

Hi Trench!

Sorry for that, it sounds like a bug – the scan should only start if you drag and drop a folder or if you use the Add Folder button. It shouldn’t scan anything for one file.

Could you please describe what exactly you did? Did you drag & drop the JPEG to the Optimizer, or did you use Add File? If we can reproduce the problem, I’ll fix it.

While optimizing renamed 17mb zip file with not zip extension (ZIP-like) after some time gave - status - exit code C0000005 , what does it mean?

Sounds like one of the Windows codes for Access Violations. Please right-click the affected file in the list, select Open log file, copy and paste it here (or send it via PM if there’s personal information in it).

[necros]

Leanify.exe --max_depth 1 "z:\TMP\papaA86BDFFD\x.zip"
Processing: z:\TMP\papaA86BDFFD\x.zip

exit code C0000005

got it - leanify error - after manual optimization it finishes with no error but file is corrupted (

[Krishty]

Leanify.exe --max_depth 1 "z:\TMP\papaA86BDFFD\x.zip"
Processing: z:\TMP\papaA86BDFFD\x.zip

exit code C0000005

got it - leanify error - after manual optimization it finishes with no error but file is corrupted (

If you send me the file, I’ll check if the problem reproduces with the latest version and if so, I’ll report the error to JayXon.

Just mentioning again that my optimizer uses a checksum to check validity of the optimized data, so even if Leanify corrupted the file silently your original would remain untouched

[Trench]

Sorry for that, it sounds like a bug – the scan should only start if you drag and drop a folder or if you use the Add Folder button. It shouldn’t scan anything for one file.

Could you please describe what exactly you did? Did you drag & drop the JPEG to the Optimizer, or did you use Add File? If we can reproduce the problem, I’ll fix it..

I dont remember how but maybe it was drag and drop or maybe I messed up.

Also I assume in the future you have a clear all button to since i usually shut it off and load it again to see have it clear.
I would be better if the instructions explained better what they do. Since one png file when I messed with the setting it increased the file size by over 900% it says from 300kb to 3mb. I think I pressed for 7zip and clear transparent.
Many of the png files seem to get stuck if selected or dragged. While slows down the bulk process.
it randomly gets stuck on jpeg images but when closing program and doing it again it works. drag and drop.
for a small 10kb gif image (20 years old file) it took over 3 minute to get to 30%. which does not do well with gif.
While this thing takes forever on small animated gif compression and doc files.
It does a good job converting other gif files to half sometimes.
hope that helps

[Krishty]

Yeah, absolutely. I guess I’ll add tooltips or help to the settings in future versions. However, every bit of UI work keeps me from implementing actual optimization.

You can fix the increased PNGs by 1) disabling PNG → for 7z 2) enabling General → Force (even if already optimized) and 3) optimizing the affected files again.

When you mention it gets stuck on JPEG randomly, does the whole program get stuck or only the JPEG optimization? There is a problem with ExifTool that causes random failures on JPEG, often with "exit code 0x000000FF" messages. I’m desperately trying to work around this using ECT because ExifTool is a huge train wreck, but it’s difficult.

DOC and ZIP will be substantially faster in a future version. There was some confusion over ECT optimizations that caused this. GIF is very aggressive and takes very long on animated images, yes. So does PNG. It doesn’t get stuck – and actually ECT just got a small improvement so it looks less frozen in my Optimizer – it optimizes down to the very last bytes. If you want less compression and more speed, use another tool, like Nikkho’s Optimizer.

[Krishty]

New version: https://papas-best.com/optimizer_en#download

added EPUB optimization

I tested the optimization on 210 ebooks. Optimization took ~5 hours with 10 threads on a Ryzen 3600. Size shrank from 135 MiB to 121 MiB, or 90.1%.

I ran the original ebooks as well as the optimized ones through epubcheck. The output before and after was identical, so my optimizations seem legit. I didn’t get any complaints over the beta version I uploaded either.

Individual compression steps FYI:

• Contained JPG is optimized. All metadata is removed (useless for EPUB), except for rotation info (since EPUB is browser-based, the layout likely builds upon rotation metadata just like normal browsers do).
• Contained PNG is optimized. All metadata is removed as is colors of fully-transparent pixels.
• Contained GIF is optimized. If not animated, it is converted to PNG. Both formats are part of the EPUB specification and readers don’t care as long as filenames and dimensions match.
• Contained XML/HTML/SVG is minified via Leanify.
• All other contained filetypes are optimized in fully lossless mode (i.e. no BMP-to-PNG conversion, ZIP timestamp removal or similar lossy tricks). The reason being, EPUB allows such non-standard types for supplemental purposes like attaching data tables to scientific papers; I won’t break what I don’t understand.
• The ZIP container is optimized according to what I explained earler.

fixed dead worker threads if errors occurred during archive handling

When processing damaged archives (ZIP, gzip, Office stuff), worker threads would fall asleep and never wake up again. This was a stupid mistake in error handling and is now fixed.

[Jaff]

1) Can optimised JPEG thumbnails be trimmed to needed size and cut the zero paddings? Thank you!
2) Option to keep JPEG trailings? (Samsung uses sometimes to store makernotes (example: J320F), interactive panorama MP4 movie (example: Galaxy S7); and aditional info in Samsung trailer format: accurate time in epoch format, panoramas info (all mobile camera photos in panorama mode), Samsung Gear 360 trailer, etc)

[Krishty]

1) Can optimised JPEG thumbnails be trimmed to needed size and cut the zero paddings? Thank you!

I currently only pass those things to ECT, so this is a huge task.

2) Option to keep JPEG trailings? (Samsung uses sometimes to store makernotes (example: J320F), interactive panorama MP4 movie (example: Galaxy S7); and aditional info in Samsung trailer format: accurate time in epoch format, panoramas info (all mobile camera photos in panorama mode), Samsung Gear 360 trailer, etc)

Yeah, I encountered this myself: https://encode.su/threads/3186-Papa%...ll=1#post62394

But now I need to start parsing JPG myself before handing it over to ECT etc., so … expect it to take quite some time.

[Jaff]

I currently only pass those things to ECT, so this is a huge task.

No problem. I found a way to fix thumbnail optimization using exiftool!
1) optimize JPEG
2) extract thumbnail into a secondary file ( exiftool -b -ThumbnailImage %1>%2 ) %1 = JPEG, %2 = thumbnail file
3) optimize thumbnail image (to cut zero padding)
4) put back thumbnail ( exiftool "-ThumbnailImage<=%1" %2 ) %1 = thumbnail file, %2 = JPEG

Done!

edit: for keeping JPEG trailers it will be much easier with the new version of exiftool - https://exiftool.org/forum/index.php?topic=10813.0

[Krishty]

Amazing hint, Jaff, thanks! To move the thumbnail back, you’ll have to put the "-ThumbnailImage<=%1" part into actual quotes.

I’ll try to quickly build that in.

[Jaff]

Put this before optimize JPEG:
0) Extract JPEG trailer to separate file (need latest exiftool 11.87)

exiftool -b -trailer 1.jpg >2.dat

5) add back the trailer:

copy /b 1.jpg +2.dat 3.jpg

1.jpg = optimised file, 2.dat (saved trailer); 3.jpg (new optimized file)
Let's rename file back and delete temp files...

del 1.jpg
del 2.dat
ren 3.jpg 1.jpg

I wait for the new version. Now you can add option to strip the trailers or not.

[Krishty]

I did some experiments with huffmix according to this post by SvenBent: https://encode.su/threads/2274-ECT-a...ll=1#post64959

(There is no public build because I haven’t gotten a response from caveman so far regarding the huffmix license.)

I tested a few thousand PNGs from my hard drive. Previous optimization used ECT + defluff + DeflOpt; now it uses huffmix on all intermediate results. Some observations:

• Without pngout, huffmix has only three streams to choose from: ECT output, ECT + DeflOpt, ECT + defluff + DeflOpt. So there is not much gain to expect.
  
  
• Actual gains were seen in about one out of fifty files. These were mostly 1-B gains; one file got 7 B smaller and another 13 B. The larger the file, the larger the gains.
  
  
• The error rate increased significantly:
  
  • “More than 1024 Deflate blocks detected, this is not handled by this version.” (known error with large PNGs)
  • “Type 0 (uncompressed) block detected, this is not handled by this version.” (known error)
  • On a few files, huffmix terminated without any error message
  
  
  
• There is a huge increase in complexity:
  
  • Previously, there was just one pipeline for all DEFLATE-based formats. I have to maintain a separate path for ZIP now, which is supported by ECT/defluff/DeflOpt but not by huffmix.
  • If huffmix exits with error code 0, all is well.
  • Else if huffmix exits with error code 1, parse stdout:
    
    • If stdout contains “Type 0 (uncompressed) block detected, this is not handled by this version.”, we have a soft error. Just pick the smaller file.
    • Else if stdout contains “More than 1024 Deflate blocks detected, this is not handled by this version.”, we have a soft error. Just pick the smaller file.
    • Else if stderr (not stdout!) contains “is an unknown file type”, we have hit a file that huffmix doesn’t understand. Just pick the smaller file. (This also happens with defluff and archives containing some Unicode characters.)
    • Else we have a hard error like a destroyed file; abort.
    
    
    
  • There is much more file I/O going on, and there seems to be a very rare race condition with Win32’s CopyFile and new processes. Not huffmix’s fault, but something that is now being triggered much more often.

All in all, I’m not sure I should keep working on it. It definitely is a great tool, but it comes with so many limitations and rough edges that the few bytes it gains me over my existing pipeline hardly justify the increased complexity.

[SvenBent]

I[*]The error rate increased significantly:

• “More than 1024 Deflate blocks detected, this is not handled by this version.” (known error with large PNGs)
• “Type 0 (uncompressed) block detected, this is not handled by this version.” (known error)
• On a few files, huffmix terminated without any error message

I can confirm both issues with huffmix and ECT on "big" files
apparently ECT just kills its with number of deflate blocks.
it was never an issue for me on pngout

I've notified the developer in his forum post about it

What I was really curious about is to do the Final ECT mixed with pngout as long as filters and palettes stay the same i guess it could be possible. but I haven't tested it much and success rate has been mixed due to the the 2 issues above

[Krishty]

I’ll try to quickly build that in.

… and then came the COVID lockdown …

I haven’t forgotten you, Jaff. I haven’t implemented the trailing data handling yet, but your thumbnail optimization suggestion works really well. Find attached the portable version of the current build; the setup/update can be found on my site.

Feedback is always welcome. I notice that JPEG optimization is somewhat slower now, because ExifTool (which is my bottleneck) runs once more per file.

Changes:

• added JPEG thumbnail optimization
• added option to remove Adobe metadata from JPEG
  This was suggested to me and it indeed strips some files further. ExifTool doesn’t do it by default. Be aware, though, that there is a good reason for that: The data may be required to correctly identify the image’s color space. If you use this option, be sure to check the result; especially if you use CMYK & Co a lot. For an example of what could go wrong, see here.
• improved launch speed
• improved performance during resizing
• updated ECT to 0.8.3
• updated ExifTool to 12.03
• fixed genetic PNG optimization appearing frozen
  Thanks to fhanau for the help!
• fixed wrong status messages
• minor performance improvements

[dado023]

Does this compresses higher than Pingo ?

[Krishty]

Does this compresses higher than Pingo ?

I don’t know If you or someone else would like to go ahead and benchmark it, I’d be very happy to learn! I’m using ECT and I’ve seen Pingo talk in the ECT thread, but to be honest I didn’t understand the bottom line.

[dado023]

Hm.....do we(ppl on forum) have standardized test collection of PNGs...i am willing to download and test, just provide me with samples to test with.

[Krishty]

Minor update and hotfix:

• ​fixed some TIFFs being destroyed during optimization
  OptiPNG’s -fix option normally helps me recover BMP/GIF/TIFF files with minor damage, but I have found that it destroys some images silently. One such image is attached.
  This may happen inside of Office documents, where the corruption is hard to find for the user. Therefore, I disabled this feature until I find a more reliable way to do it. Optimizer will now fail on such files.
• added a button to change the temp directory
  The temp directory feature never worked quite right; it is now fixed and easier to use.
• fixed temp directory not being saved correctly
  
• fixed Adobe metadata option not being saved correctly
  
• minor performance improvements
  
• minor setup improvements
  Slightly improved compression ratio on the 7-Zip download and the MSI. Small changes to the way the MSI stores paths to avoid redundancy.

The new version can, as always, be downloaded from https://papas-best.com/optimizer_en.

I’ll try to add it to the first post here as well, but this always breaks formatting, so we’ll see how it goes.

[necros]

Can new version email notification possibly be implemented in near future?

[Krishty]

Can new version email notification possibly be implemented in near future?

Good idea, that never occurred to me!

Would an RSS feed be sufficient? Mail clients like Thunderbird or Windows Mail App support RSS just like mail, and I wouldn’t have to store email addresses (better privacy).

I already maintain an RSS feed under https://papas-best.com/feed , but I could add another just for Optimizer changelogs – one notification per new version, with changes and download link.

[necros]

We have to try it, can your recommend some good RSS client for Win?