ECT, an file optimizer with fast zopfli-like deflate compression

[SvenBent]

Fixed this with ppx2 completely forgot about this godlike tool

[SvenBent]

I'm doing another big task of png optimizations and just though id share some results, now non of these are timed. For me it was more to see if we still need other tools for ultimative compresion

The list is a CHRONOLOGICAL order. its not individual compression iterations to compare
75 png files comic/drawing nature (Maps) so not reduce collors but bigger than 256

ect -9 -strip --allfilters-b    34.5 MB (36,229,177 bytes)
pngout /f6              34.5 MB (36,229,177 bytes
pngout /f5              34.5 MB (36,229,177 bytes)
pngout /f6 /b512        34.5 MB (36,229,177 bytes)
pngout /f6 /b2048       34.5 MB (36,229,177 bytes)
pngout /f6 /b8192       34.5 MB (36,229,177 bytes)
PNGOut /r (200 trials)  34.5 MB (36,229,098 bytes)
Deflopt                 34.5 MB (36,229,034 bytes)
Defluff/deflopt/huffmix 34.5 MB (36,229,015 bytes)

- Running a simple pngout with original filter or its own mixed filter did nothing for size
- Running different larger chunks sizes did nothing
- However Running 200 pngout /r trials on each file had improvements on some of the file.. Showind that pngout /r can sometimes still beats ECT -9 -strip --allfilters-b
I'm now running a new comparison where its running /r / force iteration using huffmix and deflopt and defluff in the mix for each iteration it will probably be done tomorrow

I have 2 other sets of png files that I will be testing on as wel but they are still in ECT -9 -strip --allfilters-b phase

If you have any tools you would like me to try to see if they can squeze more compression after ECT let me know and i'll give it a shoot

[MegaByte]

@SvenBent if you use this branch with options: -99999 -strip --allfilters-b --iterations=0 --stagnations=2000 --ultra=1 do you still have any files that gain improvement with PNGOut? (I still expect a bit of improvement with deflopt/defluff/huffmix).

[SvenBent]

@SvenBent if you use this branch with options: -99999 -strip --allfilters-b --iterations=0 --stagnations=2000 --ultra=1 do you still have any files that gain improvement with PNGOut? (I still expect a bit of improvement with deflopt/defluff/huffmix).

I haven't tested and im not sure i want to test it. its already taking days to go jut with -9 --allfilters-b
is stagnation the threshold amount of trials with no gains, to stop filter testing ?


This is still just on going testing but here is another test set of png. (few bigger files)

ect -9 -strip --allfilters-b    112 MB (117,451,983 bytes)
Pngout /r Huffmix (200 trials)  112 MB (118,242,599 bytes) individual files all bigger. deleted

ect -9 -strip --allfilters-b    112 MB (117,451,983 bytes)
Deflopt /b                      112 MB (117,451,983 bytes) Firs time ive seen deflopt do nothing
Defluff/deflopt/huffmix         112 MB (117,451,900 bytes)

This is the fist time i have not seen deflopt not do anyhing at all on a set of files
pngout/r huffmix is made with /force so thats why it can grow. i haev to force out a pngout generates file otherwise I cant use huffmix
I am still figuring out an easy way to handle proper pngout huffmix without risking bigger files. i need to make some comparisons in my batch script

[Scope]

I compiled the latest versions of ECT and iterations branch for Windows with folder support (the AVX2 version is also built with LTO, for some reason it shows slightly better results), but this build requires additional DLLs from MinGW and Boost.

​I also made a comparison of various PNG optimizers on a set of random public images, the goal was to get the maximum result, but in a reasonable amount of time (I did not use extreme optimization options).
https://docs.google.com/spreadsheets...gid=1741276444

ECT and Pingo, and their combination, showed the best overall result.
​Pingo showed better results due to color conversion (from 32 to 24 bits?) on large PNG (which did not do all other optimizers), but I'm not sure that it is lossless.

Examples of large PNG size differences:
Source: https://i.redd.it/02pm5dl1ipc41.png
Pingo - https://i.slow.pics/72GfqzO4.png
ECT - https://i.slow.pics/QDbFoDbW.png

Source: https://i.redd.it/mwvt7z763ol41.png
Pingo - https://i.slow.pics/7E5NXFPh.png
ECT - https://i.slow.pics/q6uXS4eb.png

[SvenBent]

​Pingo showed better results due to color conversion (from 32 to 24 bits?) on large PNG (which did not do all other optimizers), but I'm not sure that it is lossless.

In the test iran previously ect did also converter32bit (4x8bits) to 24 bits (3x8bits) images unless the alpha channel was needed
Did you check the image was 100% the same in regards to alpha channel effect as well with pingo ?

Really nice testing

[SvenBent]

Examples of large PNG size differences:
Source: https://i.redd.it/02pm5dl1ipc41.png
Pingo - https://i.slow.pics/72GfqzO4.png
ECT - https://i.slow.pics/QDbFoDbW.png

this seesm weird to me
the source is 24bit
Both ect and pingo are 48bits for some reason, without knowing to much about your test methology I would say or if this is just a weird case, i would think something went wrong in your testing

also i jsut confirmed that with 0f5oez7yvi941.png that was 32bit source got conveterd to 24bit with ect -9.
So if your testing did not show ECT converting 32bit pictures to 24 bit (As claimgi to be the benefits of pingo in your post) when appropriate, your ECT compile might have some issues in it.

-- edit --
I ran ect-9 - strip on your first ECT example ( the one with the 48bits version) and it shrunk with 94.65KB out of 7.44MB (1.2427%). im not sure if that is just meta data and your test does not count in for some optimzie removing meta data or not. but 94kb seems like a lot of meta data.. without further information i do believe the tested ect version might be incorrect
The png was still in 48 bits though


-- edit ---
I took your EVT version on you fist examepl that was in 48 bits and saved it as 24 bits.
I then shutdown my paintshop pro. reopenede both your org file and the now 24bit version of the ect file. and made a arimetihng comparison. all 0 difference on all pixels
Then I ran ect -9 -strip on it and it came down to 2.61 MB (2,741,225 bytes) down from above 7.44MB

i dont belvie ther eis any reasone to why you r 24bit source should turn into 48 bits with ect or pingo, so without further information it appears something went wrong in the testing

--- edit 3--

I retract my statement. It seems I must have confussed myself
sorcre is 48bit
ECT is 48 bits
pingo is 24 bit

Looks like ECT does not reduce 48bits to 24 bit even when the resolution is not needed ( it does do 32 to 24 though)
Maybe we cant get 48 to 24 bits and 64 to 32bit reduction implementet in ect ?

[SvenBent]

Source: https://i.redd.it/mwvt7z763ol41.png
Pingo - https://i.slow.pics/7E5NXFPh.png
ECT - https://i.slow.pics/q6uXS4eb.png

looking at source and pingo version in FF is can see a clear difference. Pingo looks more faded/grey.

tried arithmic comparios in psp7.0 they come out to be the same then i realised my PSP might not support 48bit correclt and only do a comparions on the MSB in 24bits ( alpha channels support is also weird at best)
so i used pngdiff provided to me in another thread
both source and pingo came out clean.
I then opened in windows photos and shifted back and forth and there is still a clear difference to my eyes. i do not think its placebok but have not done a propper ABX test

maybe pngdiff does not support 48bits png ?

can you confirm or deny that you see the source and pingo image as slightly difference on the mountain picture ?

-- edit ---
I used tweakpng to remove CHroma headers and other non critical chunks from the source png file leaving header idat and iend intact. now they look the same in firefox.
so the difference was maybe due to meta data removal

[Scope]

I ran ect-9 - strip on your first ECT example ( the one with the 48bits version) and it shrunk with 94.65KB out of 7.44MB (1.2427%). im not sure if that is just meta data and your test does not count in for some optimzie removing meta data or not. but 94kb seems like a lot of meta data..

I have sometimes been able to optimize some files a little bit more after reusing ECT if no extreme settings are used.
But the purpose of the test is to get the maximum efficiency in a reasonable time, with the expectation that a normal person can wait an extra few minutes to optimize a small number of files (but will not wait for hours) or put optimization overnight for a large set and that the result is ready in the morning (but waiting in weeks or months is unacceptable).

Looks like ECT does not reduce 48bits to 24 bit even when the resolution is not needed ( it does do 32 to 24 though)
Maybe we cant get 48 to 24 bits and 64 to 32bit reduction implementet in ect ?

I've already received an answer from the author on another topic: https://encode.su/threads/3108-Googl...ll=1#post64836


Although, I don't know what to do with these files for honest comparison. Remove them? Convert the original so that all optimizers do not convert colors?

[cssignet]

maybe pngdiff does not support 48bits png ?

pngdiff decodes 16 bits/sample PNGs just like web browsers, as 8 bits/sample. the difference you see comes from the iCCP chunk in the original file, deleted by pingo on the user request (aka "-strip" flag). however, the decoded raw rendered pixels by web browsers are *exactly* identical

Convert the original so that all optimizers do not convert colors?

there is *no* colors conversion from the encoder. if you load the original image in Firefox, Chromium or whatever web browser, they would convert the image data to 8 bits/sample during the decoding step without any further notice. what would be rendered is the same data that you would find in 8 bits/sample PNG optimized by pingo. if you consider this as lossy, then any 16 bits/sample PNG is lossy decoded by web browsers. FYI, pingo is not the only tool doing this. for example, the cwebp -lossless does this too for WebP

I don't know what to do with these files for honest comparison

IMHO, you should define what is the usage context. if those PNG are supposed to be used on web or not. if they do, then pingo did perform a lossless reduction. if not, you could have other issues: unlike the other tool, PNGOUT and OptiPNG are not able to perform 'alpha optimization' (which some could consider as lossy). atm they optimize RGBA samples where this transformations occurs, the comparison would be unfair

[Scope]

IMHO, you should define what is the usage context. if those PNG are supposed to be used on web or not. if they do, then pingo did perform a lossless reduction. if not, you could have other issues: unlike the other tool, PNGOUT and OptiPNG are not able to perform 'alpha optimization' (which some could consider as lossy). atm they optimize RGBA samples, the comparison would be unfair

Since this is originally random images from the web and optimization is needed for the same purpose, then most likely I will leave the same results.
I just needed to understand the reason for this big difference and how it is correct behavior, because all the other optimizers do not produce such a conversion, as opposed to alpha optimization (as it usually does not have such a large benefit of 200-300% of the file size).

[SvenBent]

pngdiff decodes 16 bits/sample PNGs just like web browsers, as 8 bits/sample. the difference you see comes from the iCCP chunk in the original file, deleted by pingo on the user request (aka "-strip" flag). however, the decoded raw rendered pixels by web browsers are *exactly* identical

Thank you for the clarification on pngdif.. I though i updated my finding by it was purely from the iCCP chunk I mist have missed in my tired head.
i used tweakpng ot remove the Iccp chunk and that was exactly the difference I was seeing visually


When you say it only decodes the 8bits/sample it means it will not detect a difference in the LSB of a 16bit /channel ?
example
Picture A has pixels value 37 F6
Picture B has pixels vale 37 F7.
pngdif would show 0 difference?

again I might have reversed my Endian format

[SvenBent]

I have sometimes been able to optimize some files a little bit more after reusing ECT if no extreme settings are used.
But the purpose of the test is to get the maximum efficiency in a reasonable time, with the expectation that a normal person can wait an extra few minutes to optimize a small number of files (but will not wait for hours) or put optimization overnight for a large set and that the result is ready in the morning (but waiting in weeks or months is unacceptable).

Nothing wrong with you comparison in my eye on that ground -9 -strip is a pretty fast and reasonable usage method. it also the one i use the most doing comparison on -9-strip- -allfilters-b --pal_sort=120 like I am doing is more for theoretically curiosity, or sometimes i just like to have my computer runs on a project for weeks. buts thats me

Although, I don't know what to do with these files for honest comparison. Remove them? Convert the original so that all optimizers do not convert colors?

My opinion on it, depends on the PNG source files
if all their pixels values are 1111010100000000 1011001000000000 0100010100000000 1000110100000000‬ i see no reason to not converting them to 8bits/channels so they are 11110101 10110010 01000101 100011010.
but if the values are not having the LSB to be all 8 0bits they should not be reduced and if pingo does that it should be noted or the comparisons, not be included.

P.S. I might have messed om my endian format but i think my point is clear. if the colors resolution is not needed converting to a lower color resolution should be fine.
just like we do with 8bits/channel picture to 8bits palette if its less then 256 colors

[SvenBent]

GOt this fancy error

Decoding error 92: too many pixels, not supported
Processed 1 file
Saved 0B out of 1.74MB (0.0000%)

What is the max size for ect to handle ?

[MegaByte]

That's coming from LodePNG.

  /*multiplication overflow possible further below. Allows up to 2^31-1 pixel

  bytes with 16-bit RGBA, the rest is room for filter bytes.*/

  if(numpixels > 268435455) CERROR_RETURN(state->error, 92);

[SvenBent]

That's coming from LodePNG.

  /*multiplication overflow possible further below. Allows up to 2^31-1 pixel

  bytes with 16-bit RGBA, the rest is room for filter bytes.*/

  if(numpixels > 268435455) CERROR_RETURN(state->error, 92);

Thank you I was at 400mill pixels 8bit rgba though

[SvenBent]

This is my final results i did run 4 different sets of png but they all contains the same type of image. these adre dungeons/room layouts for pen and paper RPG


Again please note the runs was done sequentially so a instance below another means it was executed on the output from the instance above
Some place i started over which is indicated with a blank seperation line.

--- 150dpi ---
ect -9 -strip --allfilters-b    34.5 MB (36,229,177 bytes)
pngout /f6                      34.5 MB (36,229,177 bytes
pngout /f5                      34.5 MB (36,229,177 bytes)
pngout /f6 /b512                34.5 MB (36,229,177 bytes)
pngout /f6 /b2048               34.5 MB (36,229,177 bytes)
pngout /f6 /b8192               34.5 MB (36,229,177 bytes)
PNGOut /r (200 trials)          34.5 MB (36,229,098 bytes)
Deflopt                         34.5 MB (36,229,034 bytes)
Defluff/deflopt/huffmix         34.5 MB (36,229,015 bytes)


--- Complete Floors ---
ect -9 -strip --allfilters-b    112 MB (117,451,983 bytes)
Pngout /r Huffmix (200 trials)  112 MB (118,242,599 bytes) individual files all bigger. deleted)

ect -9 -strip --allfilters-b    112 MB (117,451,983 bytes)
Deflopt /b                      112 MB (117,451,983 bytes) Firs time ive seen deflopt do nothing)
Defluff/deflopt/huffmix         112 MB (117,451,900 bytes)


--- Full --- 
ect -9 -strip --allfilters-b    109 MB (114,695,370 bytes)
Deflopt /b                      109 MB (114,695,370 bytes)
Defluff/deflopt/huffmix         109 MB (114,695,342 bytes)

ect -9 -strip --allfilters-b    109 MB (114,695,370 bytes)
Pngout /r Huffmix (200 trials)  109 MB (114,720,180 bytes)
Mixed files of the above 2      109 MB (114,628,496 bytes)


--- no grid ---
ect -9 -strip --allfilters-b    105 MB (110,486,386 bytes)
Pngout /r Huffmix (200 trials)  105 MB (110,492,868 bytes)
Mixed files of the above 2      105 MB (110,404,548 bytes)

pngout /r was done without creating a new pngout file meaning it would make anew png file if it was smaller hen the one from the previous optimizer
pngout /r huffmix was done WITH creating new files hence why it sometimes becomes bigger as a hole. its also include delftopt and defluff on each trial


conclusions in this kind of material :
pngout is still a contender and can sometimes beat out event ect -9 -strip --allfilters-b
deflopt/defluff huffmix are still nice tools to have around

[SvenBent]

Yes, please share it!

If anyone is interested, here is how Papa’s Best Optimizer does it:

1. ....
2. defluff
3. DeflOpt /bk
   • ...
   • because you said it’s best to run it after defluff
4. if defluff could not shrink the file and DeflOpt printed Number of files rewritten : 0, optimization stops here; else there has been some gain (even single-bit) and it goes back to 3.
   • This is broken in the current version and will be fixed for the next one; missed a 1-B gain on two out of 200 files.

The next time I have plenty of spare time, I want to check out Huffmix.

You really should use huffmix when using defluff and deflopt
I'll how you myt little batch part so show it it should be going

deflopt %1 /b
defluff <%1 >%1.fluff.png
deflopt %1.fluff.png /b
huffmix %1 %1.fluff.png %1

Saa bassicly you deflopt you file first.
then you create a new defluffed files
you deflop this one as well so both files are deflopt'ed
Then you use huffmix to find the optimal chunks frorm each file

The reason you do is this way is becaus deffuf breasks deflopt optimization and can create bigger files.
so this way with huffman you get the best chunks from both pre and post defluffing




This is my current new pngout scripts

@echo off
pngout %1 /force /f6


for %%f in (6,0,1,2,3,4,5) do (
for %%b in (64,96,128,192,256,384,512,768,1024,1536,2048,3072,4096,8192,16384,0) do pngout %1 /f%%f /b%%b /r
)


deflopt %1 /b


for /l %%i in (1,1,%2) do (
pngout %1 %1.tmp.png /force /f6 /ks /r
deflopt %1.tmp.png /b
huffmix %1 %1.tmp.png %1
del %1.tmp.png

defluff <%1 >%1.fluff.png
deflopt %1.fluff.png /b
huffmix %1 %1.fluff.png %1
del %1.fluff.png
)

Please not i can create bigger fiels as it forces the fist pngout file.
but this forcing a pngout fiels has to be done to be able to do all the /r trials with huffxmis
then later you have to comparee the pngout optimzied files vs ECT or whatever other you use

but bassically first part tries to find the optimal filters/chunksize combos
the second parts is just redoign it over and over with random initialization vector nad using huffmix to gain optimal chunkd from each iteration

[Krishty]

You really should use huffmix when using defluff and deflopt

While huffmix works great with pngout /r, I had little success using it on combinations of ECT/DeflOpt/defluff. Details here: https://encode.su/threads/3186-Papa%...5106#post65106

I should check whether there is a way to use ECT similar to pngout /r, i.e. whether block splits are stable with different parameters …

[SvenBent]

While huffmix works great with pngout /r, I had little success using it on combinations of ECT/DeflOpt/defluff. Details here: https://encode.su/threads/3186-Papa%...5106#post65106

I should check whether there is a way to use ECT similar to pngout /r, i.e. whether block splits are stable with different parameters …

Thank you for the testing i ran into some of the same issues with ECT. it appearsps ECT uses a lot higher number of blocks than pngout
I reported this issue to caveman in his huffmixthread
https://encode.su/threads/1313-Huffm...ll=1#post65017

Personally since Deflopt does never increase size I do not believe its has the biggest effect with huffmix
but I can ECT + defltop /b mixed with ECT+defluffed+delftop /b, as defluff sometimes increases size.

i wonder what the huffxmi succes rate is from ECT -9 with pngout /f6 /ks /kp /force on the ect file

[Scope]

since i do not have the required hardware, and when you have some time, could you try the mp in rc2 41 vs 40? -s0 on ~ 96 files should be enought (if it works - perhaps it would be possible to make it better). thanks

Unfortunately, I read it late and have already rewritten version 40, but I have done other tests with a different set:
236 files (845 416 026 bytes)

ect -1 -strip --mt-file *.png
TotalMilliseconds : 187197,7023

ppx2 -P 8 -L 1 ect.exe -1 -strip "{}" (8 MF)
TotalMilliseconds : 201311,9329

ect -5 -strip --mt-file *.png
TotalMilliseconds : 505926,7081

ppx2 -P 8 -L 1 ect.exe -5 -strip "{}" (8 MF)
TotalMilliseconds : 514763,0062

pingo (41) -s0 -strip *.png
TotalMilliseconds : 163415,1559

ppx2 -P 8 -L 1 pingo (41) -s0 -strip -nomulti "{}" (8 MF)
TotalMilliseconds : 142454,0556

pingo (41) -s5 -strip *.png
​TotalMilliseconds : 498413,4901

ppx2 -P 8 -L 1 pingo (41) -s5 -strip -nomulti "{}" (8 MF)
TotalMilliseconds : 398038,6889

pingo (42) -s0 -strip *.png
TotalMilliseconds : 126019,8598

pingo (42) -s5 -strip *.png
TotalMilliseconds : 493232,3343

[cssignet]

would you please do few more tests, preferably on the same set againts pingo rc2 43 vs proto

pingo -s0 -strip
pingo -s0 -strip -mp=8
proto -s0 -strip
proto -s0 -strip -mp=8

it would be nice if you can post the log from pingo + timer/PP64 instead of pshell. thanks

[Scope]

Same set, timer64 (Timer 14.00), pingo (43), proto.
Hmm, not bad, proto is noticeably faster and more efficient at speed 5 (including -mp=8, and faster, but slightly less efficient at speed 0 (maybe HDD in this configuration can affect, so I tested slower speed 5 too).
​
First run:

pingo.exe -s0 -strip *.png 
  pingo - (181.04s):
  -----------------------------------------------------------------
  236 files => 128508.08 KB - (15.57%) saved
  -----------------------------------------------------------------
Kernel  Time =     0.015 =    0%
User    Time =     0.000 =    0%
Process Time =     0.015 =    0%    Virtual  Memory =      9 MB
Global  Time =   181.088 =  100%    Physical Memory =      8 MB

pingo.exe -s5 -strip *.png 
  pingo - (499.17s):
  -----------------------------------------------------------------
  236 files => 148348.89 KB - (17.97%) saved
  -----------------------------------------------------------------
Kernel  Time =     0.015 =    0%
User    Time =     0.000 =    0%
Process Time =     0.015 =    0%    Virtual  Memory =      9 MB
Global  Time =   499.209 =  100%    Physical Memory =      8 MB

pingo.exe -s0 -strip *.png -mp=8 
  pingo - (122.29s):
  -----------------------------------------------------------------
  236 files => 128508.08 KB - (15.57%) saved
  -----------------------------------------------------------------
Kernel  Time =     0.015 =    0%
User    Time =     0.000 =    0%
Process Time =     0.015 =    0%    Virtual  Memory =      9 MB
Global  Time =   122.331 =  100%    Physical Memory =      8 MB

pingo.exe -s5 -strip *.png -mp=8 
  pingo - (487.98s):
  -----------------------------------------------------------------
  236 files => 148348.89 KB - (17.97%) saved
  -----------------------------------------------------------------
Kernel  Time =     0.031 =    0%
User    Time =     0.000 =    0%
Process Time =     0.031 =    0%    Virtual  Memory =      9 MB
Global  Time =   488.028 =  100%    Physical Memory =      8 MB

proto.exe -s0 -strip *.png
  proto - (97.50s): [ 8 8 ]
  -------------------------------------------------------------------------
  236 files => 128497.20 KB - (15.56%) saved
  -------------------------------------------------------------------------
Kernel  Time =     0.000 =    0%
User    Time =     0.015 =    0%
Process Time =     0.015 =    0%    Virtual  Memory =      9 MB
Global  Time =    97.553 =  100%    Physical Memory =      8 MB

proto.exe -s5 -strip *.png
  proto - (262.32s): [ 8 8 ]
  -------------------------------------------------------------------------
  236 files => 151476.87 KB - (18.35%) saved
  -------------------------------------------------------------------------
Kernel  Time =     0.031 =    0%
User    Time =     0.000 =    0%
Process Time =     0.031 =    0%    Virtual  Memory =      9 MB
Global  Time =   262.347 =  100%    Physical Memory =      8 MB

Second run:

pingo.exe -s0 -strip *.png 
  pingo - (164.35s):
  -----------------------------------------------------------------
  236 files => 128508.08 KB - (15.57%) saved
  -----------------------------------------------------------------
Kernel  Time =     0.015 =    0%
User    Time =     0.000 =    0%
Process Time =     0.015 =    0%    Virtual  Memory =      9 MB
Global  Time =   164.398 =  100%    Physical Memory =      8 MB

pingo.exe -s5 -strip *.png
  pingo - (504.40s):
  -----------------------------------------------------------------
  236 files => 148348.89 KB - (17.97%) saved
  -----------------------------------------------------------------
Kernel  Time =     0.031 =    0%
User    Time =     0.000 =    0%
Process Time =     0.031 =    0%    Virtual  Memory =      9 MB
Global  Time =   504.451 =  100%    Physical Memory =      8 MB

pingo.exe -s0 -strip *.png -mp=8
  pingo - (124.96s):
  -----------------------------------------------------------------
  236 files => 128508.08 KB - (15.57%) saved
  -----------------------------------------------------------------
Kernel  Time =     0.031 =    0%
User    Time =     0.015 =    0%
Process Time =     0.046 =    0%    Virtual  Memory =      9 MB
Global  Time =   124.992 =  100%    Physical Memory =      8 MB

pingo.exe -s5 -strip *.png -mp=8 
  pingo - (489.69s):
  -----------------------------------------------------------------
  236 files => 148348.89 KB - (17.97%) saved
  -----------------------------------------------------------------
Kernel  Time =     0.000 =    0%
User    Time =     0.015 =    0%
Process Time =     0.015 =    0%    Virtual  Memory =      9 MB
Global  Time =   489.742 =  100%    Physical Memory =      8 MB

proto.exe -s0 -strip *.png 
  proto - (97.89s): [ 8 8 ]
  -------------------------------------------------------------------------
  236 files => 128497.20 KB - (15.56%) saved
  -------------------------------------------------------------------------
Kernel  Time =     0.015 =    0%
User    Time =     0.000 =    0%
Process Time =     0.015 =    0%    Virtual  Memory =      9 MB
Global  Time =    97.936 =  100%    Physical Memory =      8 MB

proto.exe -s5 -strip *.png
  ​proto - (262.13s): [ 8 8 ]
  -------------------------------------------------------------------------
  236 files => 151476.87 KB - (18.35%) saved
  -------------------------------------------------------------------------
Kernel  Time =     0.000 =    0%
User    Time =     0.015 =    0%
Process Time =     0.015 =    0%    Virtual  Memory =      9 MB
Global  Time =   262.171 =  100%    Physical Memory =      8 MB

[cssignet]

​

proto is noticeably faster

here we are. proto is pingo, just with fair comparison of threading. thanks to your test, i would fix that later

slightly less efficient at speed 0

just chunks removal, this would be 'fixed'

now, if you are still up for it, the last trial: proto -s0 -strip, on the set you have test (~900 files) on your benchmark (heuristic test), and a comparison if possible with ECT -1 -strip. thanks!

[Scope]

​
now, if you are still up for it, the last trial: proto -s0 -strip, on the set you have test (~900 files) on your benchmark (heuristic test), and a comparison if possible with ECT -1 -strip. thanks!

I did a test on the entire PNG set (after these multithreading changes are in Pingo, I will redo the speed results of all optimizers when I have time).

498 files (2 053 230 418 bytes)

ect -1 -strip --mt-file *.png 
Processed 498 files
Saved 332.32MB out of 1.91GB (16.9717%)
Kernel  Time =     0.015 =    0%
User    Time =     0.000 =    0%
Process Time =     0.015 =    0%    Virtual  Memory =      9 MB
Global  Time =   342.581 =  100%    Physical Memory =      8 MB

proto -s0 -strip *.png   proto - (213.94s): [ 8 8 ]
  -------------------------------------------------------------------------
  498 files => 419141.53 KB - (20.90%) saved
  -------------------------------------------------------------------------
Kernel  Time =     0.015 =    0%
User    Time =     0.000 =    0%
Process Time =     0.015 =    0%    Virtual  Memory =      9 MB
Global  Time =   213.986 =  100%    Physical Memory =      8 MB

[cssignet]

​these are the results i expected. i planned changes for rc2 45 as it could require more trials. thanks for your tests, it was very useful

[cssignet]

the changes would be implemented atm (rc2 44). i did not it test widely though

[Scope]

I haven't tested the speed yet, but I've compared the overall size of the same test set (personally, the most important thing for me is maximum efficiency at the slowest speed and I would be interested in an even slower mode like -sb, but designed for large images, and at faster speeds, exchange a little efficiency for higher speed is reasonable)

498 files (2 053 230 418 bytes)

pingo (25) -s0 -strip
1 623 898 128

pingo (25) -s5 -strip
1 574 034 276

pingo (25) -sa -strip
1 555 294 498


pingo (44) -s0 -strip
1 623 898 128

pingo (44) -s5 -strip
1 577 248 424

pingo (46) -s5 -strip
1 572 187 422

pingo (47) -s5 -strip
1 570 768 051

pingo (44) -sa -strip
1 555 295 235

pingo (45) -sa -strip
1 555 294 330

pingo (46) -sa -strip
1 572 187 422

pingo (47) -sa -strip
1 555 295 745

proto -s0 -strip
1 624 029 494

proto -s5 -strip
1 567 973 360

proto -sa -strip
2 053 230 418

​

pingo (48, 49) -s5 -strip
1 570 768 051

pingo (51) -s5 -strip
1 567 820 900

pingo (48) -sa -strip
1 554 442 346

pingo (49) -sa -strip
1 554 492 020

pingo (50) -sa -strip
1 554 272 068

[cssignet]

​do not waste more time on 46, there is an error on profiles. hopefully this would be improved by speed/size in next release

edit:

pingo (50) -sa -strip
1 554 272 068

pingo (51) -s5 -strip
1 567 820 900

thanks for your test! rc3 would have same results