Papa’s Optimizer

[Krishty]

A file optimization program I wrote (Windows, 64-bit): https://papas-best.com/downloads/opt...20Optimizer.7z (7.53 MiB)

Adjust settings in the tabs, select a folder with files for optimization; press Analyze, then Optimize.

Warning:

1. Experimental – use at your own risk, always back up your files!
2. Anything DEFLATE-related can take *very* long. I don’t care; I just let it run in the background for weeks.
3. While the creation/modification date stays roughly the same, the latter is changed minimally to mark files as optimized so they are not considered in subsequent runs.
4. In case of errors, a directory with a log file and artifacts may remain at your temporary directory. I may clean it up in later versions.
5. If your username or the installation directory contains special characters, JPEG optimization may not work due to problems with EXIFTool’s Perl runtime. Sorry; I’m investigating!

Short:

This is similar to Nikkho’s File Optimizer, but the goals are a little different My optimizer tries to squeeze out every last bit without any regard to sanity. Files may take days or weeks to finish, so it’s fully multi-threaded. Under the hood, it’s the usual calls to ECT & Co. It’s a small project and I’ve only used it personally, so it doesn’t support many file formats.


Long:

I’ve been running BAT scripts on backups for quite some time, but

• the problems with unicode paths kept adding up
• good use of multiple CPU cores became more and more a concern when I cranked up compression levels
• sanity checks on the results became incredibly complicated

So I decided to write my own C++ Win32 frontend to solve these problems. I used it successfully for some years on my personal backups and projects (mostly PNG, JPG, GIF) and adjusted the UI/toolset whenever I needed to improve on anything.

The UI is inspired by Ken Silverman’s PngoutWin which I used in the 2000s. It’s multi-threaded – runs one thread per optimization, but as many parallel optimizations as you like. (Multiple threads per optimization turned out very hard to control; maybe later.)

Optimization runs with low priority, so you can keep working normally.

There’s nothing exciting to say compression-wise; all work is done by external tools. Except maybe for ArchiPNG, which I wrote a few years ago to prepare PNGs in a way they compress better with 7-Zip (especially with PPMd). No magic there, just brute-forcing filters with special zlib settings.Optimization aborts with damaged files (I think it should be up to the user to fix damaged files, not software taking educated guesses).

In general I take errors very seriously and abort to avoid data loss. However, I’ve not managed yet to enforce an actual before-after SHA check.

PNG

• Keep/delete metadata
• Full optimization via ECT
• Optimization for later archiving via a custom tool I wrote (optimal for PPMd)
• Clear/keep transparent pixels (important for textures with premultiplied alpha)
• Runs OptiPNG, ECT -60500 --allfilters-b --pal_sort=120, DeflOpt, defluff (takes very long)

JPEG

• Delete metadata, keep only Date Taken, keep anything but the thumbnail, or keep everything.
• Derotate automatically or don’t; if derotating, force losslessness or don’t.
• Runs EXIFTool for metadata, mozjpegtran for derotation, ECT for optimization.

Office (docx, xlsx, odt & Co.)

• WARNING - nonstandard! The specification requires an uncompressed MIME type as first stream in the archive, but I blindly re-compress everything. Works fine with MS/Libre Office, though.
• Recursive optimization – optimizes the contained PNG/JPEG/etc, but does not convert BMP/TGA/TIFF to PNG even if you asked for it on top-level files. (So it doesn’t destroy complex packages with specific layouts.)
• Runs 7-Zip, ECT, DeflOpt, defluff.

ZIP/gzip

• Extract, optimize, or ignore.
• Recursive optimization – optimizes the contained PNG/JPEG/etc.
• GZ optimization is a little bit weaker than Nikkho’s (because it does not remove file names).
• Runs 7-Zip, ECT, DeflOpt, defluff.

BMP/TGA/TIFF

• Can be converted to PNG (without changing the file extension).
• Runs pngout in addition to the PNG tools because it’s better at converting. It’s not permitted to re-distribute pngout, so please download it and place it in the tools subdirectory if you want to use it.

GIF

• Optimization via flexiGIF. Can take weeks.
• You can choose to convert non-animated GIFs to PNG, applying all PNG settings.

Windows

• Delete Folder.jpg *if* it is a system file, i.e. auto-generated by Windows Media Player & Co.
• Delete Album Art files from Windows Media Player.
• Delete thumbs.db.

​I’m now trying to publish my stuff instead of having it laying around on my hard drive, so it got a major rewrite last week (mostly changing the polling on sub-process output to an event-based system) and now I hope it’s useful to someone out there

[SvenBent]

I dont see pngwolf-zopfli in there for png optimizng
Also no pngout but im assuming this is due to license issues

also in my testing i have never seen optipng add anything usefull for a png optimizing routine ( ive teste it a coyple of time to see if i wanted to add it to my pngbest batch file)

P.S.
I did not know ect worked on office documents

[maadjordan]

I welcome any new tool..

I am interested in your compound file optimizer (which you did not to your Papa's Optimizer). Can you add optimizing media streams in compound files?

[maadjordan]

I did not know ect worked on office documents

Olny office 2007 and later are supported which formats falls into Zip format.

[Krishty]

I dont see pngwolf-zopfli in there for png optimizng
Also no pngout but im assuming this is due to license issues

also in my testing i have never seen optipng add anything usefull for a png optimizing routine ( ive teste it a coyple of time to see if i wanted to add it to my pngbest batch file)

P.S.
I did not know ect worked on office documents

When I last tested pngwolf, it was outperformed by ECT by a considerable margin, so I switched over to ECT entirely. That was in 2017, though …

OptiPNG is not explicitly called when optimizing PNG because it is already contained in ECT. Instead it’s used

1. to reduce color palettes & transparency before expanding PNGs (ArchiPNG doesn’t have that)
2. to convert BMP/GIF/TIFF to PNG (except for TGA, which is done via pngout) because ECT can’t do that

Yes, pngout is not contained due to its license. You can copy it into the tools folder yourself to make the TGA conversion work.

ECT works on all kinds of ZIP files, and Office documents are ZIP files (and so are ebubs). However, Office as well as epub require the first stream in the archive being the uncompressed MIME type, so running ECT on it is de-facto not allowed. It seems to work with most software, though.

[Krishty]

I welcome any new tool..

I am interested in your compound file optimizer (which you did not to your Papa's Optimizer). Can you add optimizing media streams in compound files?

Very glad that you remember Yes, the two most urgent extensions are 1. CFBF Optimizer and 2. Ultra7z. CFBF did not yet make it because I don’t have so many CFBF files; Ultra7z did not yet make it because converting that giant batch script to C is quite some hassle. It’ll come, be assured!

[Krishty]

An updated version: https://papas-best.com/downloads/opt...20Optimizer.7z

• added CFBF optimization via Papa’s Best CFBF Optimizer
• updated ECT to version 0.8.2 from 2018-08-07
• fixed inadvertent gzip optimization
• fixed leftover temporary files with recursive optimization
• fixed a crash with the context menu
• fixed a crash cancelling failed jobs
• fixed error message when archive content failed
• fixed control alignment

[CompressMaster]

@Krishty,

As to uploading, it seems to be better to upload files directly here in attachment instead of external links. Reason? When someone´s website is down either due to unpaid domain or deleted website (like disaapearance of webmaster who owned encode.ru), files will be still there.

Secondly, JPEGs are very hard to compress without partial decompression. It´s possible to make JPG compression better (more than 50%)?

Edit: General options missing - if they´re implemented.
Also, I think it will be useful if there will be detailed manual.

[CompressMaster]

Error "App not responding" when I interrupted loading files from my desktop by clicking on Cancel button.

[CompressMaster]

Error in PNG optimization (only optimize compression in PNG was ticked) - ECT_x64.exe -60500 --allfilters-b --pal_sort=120--strict "C:\users\acer\AppData\Local\Temp\c1cDC3E.tmp\x.pn g" It takes too long - actually, it´s not completed (I´m waiting 5 minutes and nothing happens).

File is attached.

Also, it will be useful if there will be progress bar indication.

[Krishty]

As to uploading, it seems to be better to upload files directly here in attachment instead of external links. Reason? When someone´s website is down either due to unpaid domain or deleted website (like disaapearance of webmaster who owned encode.ru), files will be still there.

Right, I added an attachment!

Secondly, JPEGs are very hard to compress without partial decompression. It´s possible to make JPG compression better (more than 50%)?

I don’t have added a JPEG filter, only one for PNG.

Also, I think it will be useful if there will be detailed manual.

Coming, but no time yet.

Error "App not responding" when I interrupted loading files from my desktop by clicking on Cancel button.

I’ll check that!

Error in PNG optimization (only optimize compression in PNG was ticked) - ECT_x64.exe -60500 --allfilters-b --pal_sort=120--strict "C:\users\acer\AppData\Local\Temp\c1cDC3E.tmp\x.pn g" It takes too long - actually, it´s not completed (I´m waiting 5 minutes and nothing happens).

Read the first post, 2nd Warning: Anything DEFLATE-related can take *very* long. I don’t care; I just let it run in the background for weeks.
If you just want a quick optimization and don’t care about the last few bytes, Nikko’s File Optimizer finishes much faster, I think.

Also, it will be useful if there will be progress bar indication.

Not possible because ECT has none

[CompressMaster]

I want to test maximum compression corpus, but MCC (from folder) cannot be loaded at all. I thought that there´s problem with unsupported filetypes or folder path with strange characters such as C:\user\acer\desktop\test folder\čížšťčžš$ßß÷\, but there´s another problem that I am unable to indentify. Other files are OK.

[Krishty]

Works for me, even if I put it into a folder named A:\Users\Krishty\Desktop\corpus\čížšťčžš$� �ß÷.

• A10.jpg: -71237 B; -8 %
• ohs.doc: -10752 B; 0 %
• rafale.bmp: didn’t wait for finish

If you can isolate the issue, I’d be very glad to hear about it!

[SvenBent]

When I last tested pngwolf, it was outperformed by ECT by a considerable margin, so I switched over to ECT entirely. That was in 2017, though …

OptiPNG is not explicitly called when optimizing PNG because it is already contained in ECT. Instead it’s used

1. to reduce color palettes & transparency before expanding PNGs (ArchiPNG doesn’t have that)
2. to convert BMP/GIF/TIFF to PNG (except for TGA, which is done via pngout) because ECT can’t do that

Yes, pngout is not contained due to its license. You can copy it into the tools folder yourself to make the TGA conversion work.

ECT works on all kinds of ZIP files, and Office documents are ZIP files (and so are ebubs). However, Office as well as epub require the first stream in the archive being the uncompressed MIME type, so running ECT on it is de-facto not allowed. It seems to work with most software, though.

When i tested ECT -9 when it first came out it didn ad alot compared to running pngout ang pngwolf-zopfli.
pngwolf did alot more delta fitlering coparions back then.
Im not sure its the same now or if it you are running ect even higher than -9 and that why pngwolf-zopfli is not adding a lot.

Just given back my 2 cents. I'll see if i can find the test results. even thoug they are probably horrible outdated

[nikkho]

Looking very nice.

[necros]

is BMP, GIF to PNG conversion lossless?

[Krishty]

For pixels, yes. Metadata not.

[CompressMaster]

@Krishty,
1,By attaching, I mean your 1st post. Could you repair that? Thanks.
2,What other unpublished stuffs do you have? (compression field)

[Krishty]

1,By attaching, I mean your 1st post. Could you repair that? Thanks.
2,What other unpublished stuffs do you have? (compression field)

Fixed.

A few years ago, I wrote a custom PNG variation with PPMd instead of Deflate which worked pretty well with the expand for 7z function in my optimizer. However, I ditched it because non-standard formats are pretty much useless. Now I’m investigating ECT’s efficiency. Nothing else comes to my mind right know.

---

The Optimizer has a (non-critical) memory problem with GIF optimization. FlexiGIF outputs a *lot* of progress information, sometimes as much as a GiB over a few days of run-time. Optimizer keeps all that (needlessly) in memory. I’ll fix that for the next version.

[Krishty]

Another update in the first post or on https://papas-best.com/downloads/opt...20Optimizer.7z

• improved ECT PNG speed: three blocksplitting passes (six before); 60 iterations (500 before)
• added version information to EXE
• fixed memory leak with huge log files (e.g. during GIF optimization)
• fixed glitch in ECT command line

[necros]

Feat. req. - Drag`n`drop support

[Krishty]

Absolutely agreed, already on my internal todo!

[CompressMaster]

In case of errors, a directory with a log file and artifacts may remain at your temporary directory. I may clean it up in later versions.

Better approach would be to store log files (they can be very useful for further investigation of errors!) alongside with original file - e.g. original.png - original.log in folder where original files are.

If your username or the installation directory contains special characters, JPEG optimization may not work due to problems with EXIFTool’s Perl runtime. Sorry; I’m investigating!

This can be solved relatively easy. Just temporarily copy each processed file for example at C:\PapaOptimizer\, process it here and finally copy output at the desired location.

My optimizer tries to squeeze out every last bit without any regard to sanity.

That´s great! Best possible lossless compression is always welcomed!

Optimization runs with low priority, so you can keep working normally.

That´s OK, but it will be good if priority can be specified manually in general options. Some users (including me) might need best possible compression with high CPU usage regardless time.

JPEG
ECT for optimization.

Can you be more specific?

Also, it will be good if debug infos from CMD software´s can be seen at debug window in GUI.

[Krishty]

Better approach would be to store log files (they can be very useful for further investigation of errors!) alongside with original file - e.g. original.png - original.log in folder where original files are.

Nah, this way you pollute directories if something goes wrong. In its current form, you can open the log file and save it wherever you like.

This can be solved relatively easy. Just temporarily copy each processed file for example at C:\PapaOptimizer\, process it here and finally copy output at the desired location.

Creating arbitrary directories is bad practice. MSI does it and everyone hates it (and rightly so). Furthermore, you don’t have write access to the system drive since Vista. %TEMP% is the only directory for that stuff.

I could force the user to choose a different temporary folder if the current one contains special characters. But even better would be fixing the Perl problem in the first place! I’ll check if locale settings change anything for I suspect the runtime gets the strings from the process startup structure wrong.

That´s OK, but it will be good if priority can be specified manually in general options. Some users (including me) might need best possible compression with high CPU usage regardless time.

Time for a General options tab, I guess …

Can you be more specific?

Also, it will be good if debug infos from CMD software´s can be seen at debug window in GUI.

Check this out: https://encode.su/threads/1589-FileO...ll=1#post61621

I don’t see what a debug window would be good for. You can always check the logs!

[JayXon]

I noticed you are using Leanify with --max_depth 1 on office files and I'm wondering why? Leanify can optimize xml in office files, which doesn't save much but still better than not optimize it.
For example, with this pecoff.docx (attached), Leanify with default option optimized it to 178,035 bytes in seconds while Best Optimizer optimized it to 178,044 bytes and took very very long time.

[Krishty]

I noticed you are using Leanify with --max_depth 1 on office files and I'm wondering why? Leanify can optimize xml in office files, which doesn't save much but still better than not optimize it. For example, with this pecoff.docx (attached), Leanify with default option optimized it to 178,035 bytes in seconds while Best Optimizer optimized it to 178,044 bytes and took very very long time.

Hi JayXon! First of all, thank you for your great software and for testing mine!

The main rationale behind --max_depth 1 decision was: ECT optimizes PNGs better than Leanify (see here), so Leanify should not waste its time on PNGs that it cannot make more compact anyway.

Proof: I can reproduce your results if I tick Office optimization + content optimization. But I get even higher compression (177,997 B) when I go to the PNG tab and tick Clear transparent (Premultiplied) pixels / Delete metadata.

But there are other issues here:

• You are correct that I am missing out on XML, and that’s bad (I reach 177,880 B combining Papa’s PNG with Leanify’s XML). It’s because my Optimizer does not (yet) correctly tell apart Office files from ordinary ZIPs. I don’t want to change XML files in some programmer’s personal ZIP backup, so I set --max_depth 1 for safety, unwillingly affecting Office files as well. I’ll change that in a future version once I detect Office files reliably.
  
  
• However, for that I need a way to disable specifically PNG/JPEG optimization in Leanify (or else it will be done twice with no benefit over ECT, maybe even deleting metadata that was meant to be kept). Could you help me there?
  
  
• The slowness comes from problems with ECT. While ECT is totally superior to everything else on PNG and JPEG, I noticed severe problems with ZIP. I had to hot-fix it poorly, resulting in bad performance. I’ll fix it properly and improve ZIP performance after I finish my PNG benchmarks.
  
  
• My options to clear transparency and metadata on PNG should always be enabled for data in Office files. It’s clearly not something that the user should have to pick manually. This, too, requires correct Office detection.

[Krishty]

JayXon, thanks to your test file I found a bug in the old ECT version I use. (You may notice that the image in the DOCX breaks sometimes.)

The bug has been fixed in ECT in 2018, i.e. this build is fine. The reason I didn’t use it is that I had used (i.e., tested) the 2017 version extensively and didn’t want to introduce new bugs with an update (how ironic).

@all: There is a Bug in Zip/Office Optimization which can result in DATA LOSS. Please update to the latest version!

If someone wants to test their ECT or something, check the reduced test case I attached. ECT -9 --disable-png -zip can save a byte on that Zip, but the buggy version cuts off one byte from the beginning of the PNG, breaking the file. It’s not an ECT problem any more; it’s me using an outdated version.

[JayXon]

Hi JayXon! First of all, thank you for your great software and for testing mine!

The main rationale behind --max_depth 1 decision was: ECT optimizes PNGs better than Leanify (see here), so Leanify should not waste its time on PNGs that it cannot make more compact anyway.

Proof: I can reproduce your results if I tick Office optimization + content optimization. But I get even higher compression (177,997 B) when I go to the PNG tab and tick Clear transparent (Premultiplied) pixels / Delete metadata.

But there are other issues here:

• You are correct that I am missing out on XML, and that’s bad (I reach 177,880 B combining Papa’s PNG with Leanify’s XML). It’s because my Optimizer does not (yet) correctly tell apart Office files from ordinary ZIPs. I don’t want to change XML files in some programmer’s personal ZIP backup, so I set --max_depth 1 for safety, unwillingly affecting Office files as well. I’ll change that in a future version once I detect Office files reliably.
• However, for that I need a way to disable specifically PNG/JPEG optimization in Leanify (or else it will be done twice with no benefit over ECT, maybe even deleting metadata that was meant to be kept). Could you help me there?
• The slowness comes from problems with ECT. While ECT is totally superior to everything else on PNG and JPEG, I noticed severe problems with ZIP. I had to hot-fix it poorly, resulting in bad performance. I’ll fix it properly and improve ZIP performance after I finish my PNG benchmarks.
• My options to clear transparency and metadata on PNG should always be enabled for data in Office files. It’s clearly not something that the user should have to pick manually. This, too, requires correct Office detection.

Hmm, it didn't occur to me that options in other tab can affect office file too.


How about flags to disable specific format? For example --disable-png.
Feel free to send me pull requests if you want this asap, I probably won't get to do it myself anytime soon.

[Krishty]

Updated version: https://papas-best.com/downloads/opt...20Optimizer.7z

Changes:

• improved Office optimization (if you already optimized your Office documents with an earlier version, check the Force option in the bottom-left corner to re-optimize)
• updated ECT to version 0.8.2 from 2018-08-07 (bugfix for ZIP compression)
• jobs can now be cancelled (via right-click context menu)
• fixed JPEG optimization failure if derotation would be lossy
• fixed recursive gzip optimization
• fixed initial value of “Compound” checkbox
• fixed exit code message on tool errors
• added icon
• improved layout of archive tabs
• numbers now display localized

The analysis step got a major overhaul. First and foremost, this results in better compression of Office files (optimization, conversion, and derotation of PNG/JPEG/GIF/TIFF inside is now done using pre-tuned settings; XML is now minified using Leanify). Furthermore, it protects you from shooting yourself in the foot by accidentially derotating a JPEG inside an Office file, thereby breaking it – JPEG derotation is simply ignored inside of Office files now.

With Office files I mean all of Microsoft’s DOCX, PPTX, … (commonly referred to as Office Open XML); Open Office’s ODT, ODS, … (Open Document) and also XPS and related files.

Though I tested it thoroughly, be sure to back up your files and check the results. Major overhauls often bring problems, especially in such a complex ecosystem with so many tools and file formats.


@JayXon: I’d prefer a whitelisting model over a blacklisting model, e.g. “disable anything but XML”. For now, I’m fine with calling Leanify on individual files with --max_depth 1. One thing I’d like to backport from ECT, though, is an option to keep filenames inside GZ archives. I’ll send a pull request once I get around doing that.

Now that I use Leanify on XML, pecoff.docx compresses to 177,916 B. The 177,880 I got earlier were due to data loss with the ECT bug. But still that’s a great improvement IMHO. Run-time is still annoyingly long, but I’ll improve it with one of the upcoming versions

[CompressMaster]

As to uploading, it seems to be better to upload files directly here in attachment instead of external links. Reason? When someone´s website is down either due to unpaid domain or deleted website (like disaapearance of webmaster who owned encode.ru), files will be still there.

Krishty, why you uploaded new version in form of external links? It seems to be better to upload each version directly in attachments as I explained earlier.