Papa’s Optimizer

[Krishty]

Krishty, why you uploaded new version in form of external links? It seems to be better to upload each version directly in attachments as I explained earlier.

But I did! Check the attachment in the first post

[dado023]

I am trying to downsize larger gif, but Papa's is doing to already 4 hours or so.

Is there any way to use papa optimizer with less time requirements but still size it down?...or even with lossy compression.

For testing purposes file is in 7zip in attachment

[Krishty]

Yeah, GIFs can take months to complete and it makes sense considering that I go for maximal compression. Speeding this up is a TODO and I’m strictly lossless, so no luck right now.

You could try Nikkho’s File Optimizer, or open a command window in the tools directory and directly call gifsicle -o out.gif in.gif. But your GIF seems pretty well-optimized already, so don’t expect large returns.

[necros]

I know main goal is max compression, but Fast and Normal options would be great for quick testing / casual usage.

[ASkript]

Hi!
I tried to optimize an odt file for test. App just sopped responding after extracting content, but at the same time CPU is idling. I'm using Windows 10 build 18362.356

[Krishty]

Thanks for testing and very sorry for the problems you experience! Can you please help me investigate via the following:
1. When the app stopped responding, open Task Manager -> Details -> Right-click Best Optimizer.exe -> Create Dump -> send the file to me (info at papas-best.com), maybe zipped.
2. Clear your %TEMP% directory (C:\Users\ASkript\Appdata\Local\Temp) from any sub-directories starting with "c1c". Start the optimization again. When the app hangs, there should be a new folder starting with "c1c" and it should contain a log.txt file. Please send this file as well.

[Krishty]

Updated version with drag’n’drop support and optional foreground priority: https://papas-best.com/downloads/opt...20Optimizer.7z

This was a major UI overhaul, so regard this as a beta release.

• major UI overhaul (now supporting drag’n’drop)
• moved options to new General tab
• fixed 7-Zip settings
• fixed crashes with analysis
• fixed a typo
• fixed tab flickering

[JamesWasil]

Papa's Optimizer: Better Ingredients. Better Refinements. Papa Op's.

Once the optimizations can be delivered under 30 minutes with a Papa Tracker, I would tip for that.

[Krishty]

Hah! I’ll see what I can do

[Krishty]

Dear ladies and gentleman,

another update attached to the first post or downloadable via https://papas-best.com/downloads/opt...20Optimizer.7z .

It is, first and foremost, a bugfix release:

• options are now disabled during optimization
• fixed possible deadlock with Leanify
• fixed possible deadlock with damaged archives
• fixed missing 7z.dll
• fixed launch error message
• fixed tab glitches
• fixed keyboard navigation

@ASkript: Bugs have been fixed that only surfaced on my freshly installed Windows 10. There’s a chance that it works better for you now.

For those who haven’t seen the reworked UI yet:



The UI will now flicker with the Windows 7 Classic Theme during resizing, but that’s the price you have to pay for correct tab rendering on Windows 10. (If anyone is interested in Win32 tab control details, I wrote about the changes in German here).

Please let me know if I broke anything!

[necros]

Considered to upload it to Github?

[Krishty]

Considered to upload it to Github?

Yes, but only after further code clean-up. When I started the program a few years ago, it was just meant as a replacement for my Batch scripts, with no error handling and no public distribution in mind, and entangled with my other projects. It’ll take a while to resolve that.

If you want to play around, PM me and I’ll send you a 7z with the Visual Studio project.

P.S.: First build from 2015; not so much change after all

[necros]

P.S. where did you get archipng utility? Can`t find source

[Krishty]

I wrote it myself, in 2009 or so.

It just runs 32 trials with each filter on or off (2^5) and uses zlib with Z_FIXED compression to determine the smallest result. Then it writes this filter combination without any compression.

IIRC, Z_FIXED behaves like a pretty good approximation for PPMd; better than the other modes. So you will probably get the best results compressing these PNGs with PPMd afterwards.

But keep in mind, it has been ten years since I worked on it so I’m happy to learn other tools doing better jobs

Edit: Attached the only copy of the source code I have.

[Krishty]

Some development notes in case anyone is interested.

Data integrity is paramount; there is nothing worse than finding out your backups are corrupted when you desperately need them.

I want my Optimizer to perform an integrity check on optimized data to find out whether something broke. Since rumors go around that defluff may corrupt archives (though I cannot reproduce this specific case), I decided to start checking the results of ZIP/gzip/Office optimization.

I imagined I’d call 7-Zip’s checksum feature before and after Deflate optimization and compare the output. Should be simple enough.

But it’s not. As I noticed quickly, 7-Zip treats archives with just one file in them as a special case – it only computes a checksum of data then, but not of names, to optimize for information size.

After working around this, checksums failed on almost all ZIP files I tried. The reason is the ZIP central directory after optimization with ECT/defluff/DeflOpt does not contain directory records any more. Directory names are instead coded directly into file names as relative paths (a size optimization, obviously). 7-Zip does not consider this case during checksum computation. I reported this to Igor Pavlov, but I don’t know yet whether this optimization is legal in the first place.

You’ll probably have to wait for a new version until I worked around all this and tested it thoroughly.

[Krishty]

I’ve released a new version. See https://papas-best.com/optimizer_en#download or attachments to first post.

Changes:

• added checksum test to archive optimization
  I want to avoid silent data corruption at any cost. For ZIP/gzip, the Optimizer now uses 7-Zip to compute checksums of archive content before and after optimization with AdvZIP/ECT/DeflOpt/defluff. If data is corrupted, the job will abort with the message ERROR: content broke during optimization and the original file remains untouched.
  I noticed that ECT does not have any means to disable recursive optimization. I.e. if you optimize a ZIP which contains another ZIP, and you unchecked recursive, then optimization will fail with said message because ECT optimized with the inner ZIP file even though you didn’t want that. I still need to decide how this situation should be treated gracefully.
  Names will not be checked. I would have loved to do this, but:
  • DeflOpt deletes all empty directories from archives, changing the checksum. I could work around that, but … it takes time.
  • 7-Zip has a bug with checksums of file names in ZIPs if they contain relative directories (as commonly produced by DeflOpt/defluff).
  • 7-Zip cannot compute name checksums for single items (as an information size optimization).
  
  
• added setup
  You can now download a MSI package of the optimizer. It’s plain simple; no choosing paths or anything. By default, it installs for the current user only. If you want to install it for all users, run msiexec /i "Papas Best Optimizer.msi" MSIINSTALLPERUSER=""
  
  
• fixed mislabeled Cancel button
  There were two Optimize buttons instead of Optimize and Cancel. Stupid copy-paste error during UI cleanup. Fixed.
  
  
• fixed one thread appearing dead in the UI
  If your CPU had four cores and you selected four threads, the 4th thread would appear stuck even though it optimized fine. Fixed that.
  
  
• fixed recursive ZIP setting
  The recursive check often did not work. Fixed that.
  
  
• fixed gzip optimization potentially deleting filenames
  There was a formatting bug in ECT’s command line, hiding the --strict switch for gzip archives which normally forces preserving file names. My tests did not produce any errors, but it is possible that some gzip files lost file names. Anyway, that’s fixed now.

[OneDeltaTenTango]

Did some quick tests. I really like the multi threading.
Its not APNG safe, test file https://red.dazoe.net/stuff/9.png
Just to see what happens, I threw my comics/manga at it and will its still Analyzing after +8hr but still progressing. granted its like ~75,000 files.
On the other-hand FileOptimizer from https://nikkhokkho.sourceforge.io/st...=FileOptimizer would at-least be working on it by now, sadly its not multi threaded and Pingo from https://css-ig.net/ would have been done with it by now if it managed to not hit a folder with a super long path or strange characters.

If i didn't miss it somewhere would it be possible to add an endless trials/brute-force compression option so it endlessly loops on a batch of pngs.

[Krishty]

Glad you like it!

Its not APNG safe, test file https://red.dazoe.net/stuff/9.png

Excellent find! I’ll see what I can do.

Just to see what happens, I threw my comics/manga at it and will its still Analyzing after +8hr but still progressing. granted its like ~75,000 files.

Nonetheless, that’s too long. I recommend:

1. Throw a dozen of files at it so Optimize becomes enabled and click that. Really just a few files to get it going with all available threads. Then throw your huge directories at it and it will analyze in the background while optimizing. (I may fix that clunky handling in a future version.)
2. I know Windows has some limitation where you cannot create more than 65k temporary directories. My Optimizer may well be affected. Better run it with batches of 10k until I reworked the temporary directory handling.

If i didn't miss it somewhere would it be possible to add an endless trials/brute-force compression option so it endlessly loops on a batch of pngs.

You will notice PNG takes extremely long already, so be careful what you wish for Infinite looping will not buy you anything, I’m afraid. The reason is in the curves here: https://encode.su/threads/2274-ECT-a...ll=1#post61627
I’m obsessed with squeezing out single bytes, rest assured, but we’re within 100 B of the perfect result using the current tools and it takes other means to improve compression. E.g. I’ll try running huffmix on a collection of ECT passes soon.

[suryakandau@yahoo.co.id]

Glad you like it!

Excellent find! I’ll see what I can do.

Nonetheless, that’s too long. I recommend:

1. Throw a dozen of files at it so Optimize becomes enabled and click that. Really just a few files to get it going with all available threads. Then throw your huge directories at it and it will analyze in the background while optimizing. (I may fix that clunky handling in a future version.)
2. I know Windows has some limitation where you cannot create more than 65k temporary directories. My Optimizer may well be affected. Better run it with batches of 10k until I reworked the temporary directory handling.

You will notice PNG takes extremely long already, so be careful what you wish for Infinite looping will not buy you anything, I’m afraid. The reason is in the curves here: https://encode.su/threads/2274-ECT-a...ll=1#post61627
I’m obsessed with squeezing out single bytes, rest assured, but we’re within 100 B of the perfect result using the current tools and it takes other means to improve compression. E.g. I’ll try running huffmix on a collection of ECT passes soon.

https://youtu.be/eh4c0prEU7g
How extremely long for PNG file ? 1 day ? 1 week ?

[Krishty]

What’s the relation of your link to the topic?

My CPU is old, so your mileage may vary:

RGBA 100²: a minute or so
1000² pixels: 1–2 hours
for palettized images: multiply with 120 due to ECT trying 120 palette permutations

At least it scales very well with the number of cores.

[suryakandau@yahoo.co.id]

What’s the relation of your link to the topic?

My CPU is old, so your mileage may vary:

RGBA 100²: a minute or so
1000² pixels: 1–2 hours
for palettized images: multiply with 120 due to ECT trying 120 palette permutations

At least it scales very well with the number of cores.

I just want to show that android can do computer vision not too long. I think computer vision and data compression has the related process. Like what or which byte to recognize etc.

[Krishty]

Annotated changelog of the latest version from https://papas-best.com/optimizer_en (testing is welcome as always):

optimization now starts immediately
No more analyze/optimize separation; should make it easier to use

fixed gzip inadvertently being converted to ZIP
Due to an error in a command line, some gzip archives were sometimes converted to ZIP by accident. If you used the previous version to optimize gz archives and you care about the format, please check the optimized files.

fixed handle exhaustion due to log file leaks
fixed cleanup of temporary directories
There was a leak in log file management. It surfaced as seemingly random failures or crashes when optimizing thousands of files and may have spammed your temp directory.

new multi-threading
The threading has been completely reworked. The number of threads is now selected in the bottom-right instead via options.

• You can change the number of threads during optimization. (But consider that lowering it will not take effect immediately because no jobs are being forcefully canceled.)
• You can pause the optimization by selecting zero threads.
• Countless scheduling bugs have been fixed, e.g. a DOC optimization expanding to multiple sub-optimizations not being multi-threaded.
• Optimization starts with the first file you add and runs parallel with further analysis.
• It’s also a very minimal amount faster because the new implementation has less locking than the old one. (It is now based on a producer-consumer pattern with a thread pool and condition variables.)

items can now be canceled before optimization started
cancelation now considers sub-items
… because that kept annoying me.

fixed optimized files sometimes being timestamped older than originals
fixed a crash after optimizing many files
Wrong math and a null pointer access.

Sorry that I didn’t get to APNG yet. I have a similar problem with JPG files where some cameras stuff multiple images into the same file, but my Optimizer deletes all but the first. I’m looking into it.

[necros]

Can you show example of such jpg which stuff 2 images in single file?

[Krishty]

Can you show example of such jpg which stuff 2 images in single file?

Sure thing! I attached two samples, both taken by friends of mine.

One is an MPO renamed to JPG. MPO is Multiple Picture Object and it’s an extension to the JPG standard. It basically stores two JPGs for stereoscopic display. Using my optimizer on this file reduces its size to 49%, mostly so by just deleting the other image, which is very unfortunate. At least the format is well-documented, so with some effort I might be able to handle it properly.

The other file is something I don’t understand at all. It’s a JPEG taken with a Huawei phone. At 3968×2976 (12 MP), it amounts to 18,2 MiB(!!!). If I run it through my optimizer, it reduces to 3,2 MiB (the expected size). The thing is – I have no idea what’s been deleted. My friend says, it’s a feature to share “moments” instead of just pictures. If anybody could tell me what’s the thing with this file, I’d be glad to know!

[schnaader]

Sure thing! I attached two samples, both taken by friends of mine.

The first one seems to be just a matter of parsing as the MPO format just is the concatenation of two standard JPGs (with metainformation in APP2 markers) - if treated as a file with two embedded JPGs, it should work. Precomp does this and detects the 2 JPG streams successfully:

Possible JPG found at position 0, length 619813
Possible JPG found at position 619813, length 563230

// note: 619813 + 563230 = 1183043 which is the size of the file

The second one is JPG data (position 0, length 3677374) followed by some header data (~5 KB). After that, "ftypmp42" indicates a video which can be extracted (beginning at offset 0x3831EA) and played (tested with VLC). So it's basically a short video (2 seconds, 960x720 H264 video, AAC audio) from the moment the picture was taken - attached it to the post. There could be some data after it, too, but shouldn't be too much if any.

[Krishty]

Confirmed! Tausend Dank, schnaader!

I think I need to start dissecting JPG files before optimizing them …

[Krishty]

New version: https://papas-best.com/optimizer_en

added SVG optimization via Leanify
Not much to say here. Only supporting “real” SVG so far, no SVG embedded in XML & Co.

removed TGA-to-PNG support (due to PNGOUT license)
This never worked in the public build anyway, so I removed it.

fixed crashes
Stupid little synchronization issues. They crashed the Optimizer especially when optimizing many small files in rapid succession.

fixed PNG optimization sometimes missing a 1-B opportunity
The DeflOpt/defluff loop did not work properly when gains were below 1 B. On 250 files of mine, the new version improved compression on three. The gains were 1, 1, and 3 bytes. No big deal, but the improvement comes with virtually no cost.

fixed very small window dimensions
You can no longer resize the UI so small that it looks stupid.

[necros]

Hm, RMB click on file llist area exits Papa`s, bug? (win10 x64)
Feat. req.- Auto save settings

[Krishty]

Hm, RMB click on file llist area exits Papa`s, bug? (win10 x64)Feat. req.- Auto save settings

Definitely a bug. It doesn’t happen if you right-click a file name directly, only on the blank area (I hope!). Will be fixed, thanks!

[Krishty]

necros, the bug is fixed in this version: https://papas-best.com/optimizer_en

Thank you again for reporting it!