ECT, an file optimizer with fast zopfli-like deflate compression

[nikkho]

Fixed.

Thanks for the x64 build.

[przemoc]

I've built it on my W7 x64 in up-to-date MSYS2's MinGW-w64 Win64 Shell using recent:

gcc version 5.3.0 (Rev1, Built by MSYS2 project)

after performing

( cd src/mozjpeg/ && autoreconf -fi )

adding LDFLAGS to src/Makefile:

LDFLAGS= -static -static-libgcc -static-libstdc++

and using these LDFLAGS during linking.
Binary in the attachment is stripped.

Seems to be working fine with recent fix.

Felix, don't you think using

#if defined(...) && !defined(...) && ...

would be a bit nicer than 3 levels of #ifdef hell?

[przemoc]

New build using latest commit (with fixed Makefile).

Felix, you forgot to update Makefile after removing zlibWrapper.

[lorents17]

I made two test, here results:

test 1 (8600 PNG):
iCatalyst - Advanced - 604 846 779 byte - 0:42:13
ect from 25/12/2015 - M2 - 604 383 889 byte - 0:37:03
ect from 01/01/2015 - M2 - 605 726 841 byte - 0:34:14

test 2 (3800 PNG):
iCatalyst - Advanced - 1 184 876 290 byte - 0:47:16
ect from 25/12/2015 - M2 - 1 181 220 466 byte - 0:42:46
ect from 01/01/2015 - M2 - 1 184 451 798 byte - 0:38:42

[fhanau]

I made two test, here results:

test 1 (8600 PNG):
iCatalyst - Advanced - 604 846 779 byte - 0:42:13
ect from 25/12/2015 - M2 - 604 383 889 byte - 0:37:03
ect from 01/01/2015 - M2 - 605 726 841 byte - 0:34:14

test 2 (3800 PNG):
iCatalyst - Advanced - 1 184 876 290 byte - 0:47:16
ect from 25/12/2015 - M2 - 1 181 220 466 byte - 0:42:46
ect from 01/01/2015 - M2 - 1 184 451 798 byte - 0:38:42

This happened probably due to this and this. These commits improve speed and text compression, but reduce PNG compression as blocksplitting and costmodel get worse. But I also made improvements since 25/12 and the compression on my own test images actually got better.

[lorents17]

New test:

test 1 (8600 PNG):
iCatalyst - Xtreme - 598 100 342 byte - 2:42:52
ect by 25/12/2015 - M4 - 598 402 908 byte - 2:49:57
ect by 01/01/2015 - M4 - 599 312 198 byte - 2:33:20

test 2 (3800 PNG):
iCatalyst - Xtreme - 1 170 480 787 byte - 3:52:29
ect by 25/12/2015 - M4 - 1 170 850 232 byte - 4:48:48
ect by 01/01/2015 - M4 - 1 172 646 702 byte - 4:23:33

[lorents17]

test 1 (8600 PNG):
ect by 06/01/2015 - M2 - 604 801 930 byte - 0:34:22
ect by 06/01/2015 - M4 - 599 303 054 byte - 1:05:31

test 2 (3800 PNG):
ect by 06/01/2015 - M2 - 1 182 829 456 byte - 0:39:41
ect by 06/01/2015 - M4 - 1 172 638 358 byte - 1:30:58

Results of the last dough simply impress, especially the mode - M4
Now I test the new version.

[fhanau]

I released Version 0.2.
Changes:
Nine modes, usage is now -1 to -9 instead of -M1 to -M5
Much higher speed when using many iterations as I added an matches cache. lorents, this caused your speed improvements.

New stats on enwik8 (gcc 5.1, Intel i5-2500S @ 2.7Ghz, OS X 10.10):

[lorents17]

fhanau
I very much ask you, make if it is possible, the zopfli version compatible to the original version.
And it is even better to make patches, for the original zopfli version and to send to developers.

[nikkho]

Thanks. Could you post on the releases page, also the win32 one?

[lorents17]

test 1 (8600 PNG):
ect -2    607 423 500    0:35:23
ect -3    604 707 780    0:38:49
ect -4    605 112 805    0:39:18
ect -5    602 818 906    0:44:29
ect -6    600 773 213    0:51:02
ect -7    599 199 736    1:01:22
ect -8    597 995 893    2:12:37
ect -9    597 985 702    2:23:06

test 1 (3800 PNG):
ect -2    1 190 882 178    0:34:27
ect -3    1 182 829 456    0:43:23
ect -4    1 184 709 350    0:46:37
ect -5    1 180 032 491    0:55:41
ect -6    1 175 521 282    1:06:52
ect -7    1 172 470 148    1:31:28
ect -8    1 170 570 598    3:29:24
ect -9    1 170 570 598    3:42:33

It is very interesting to test your zopfli version with a combination of TruePNG and PNGWofl. It is sure results of compression it is possible to increase significantly.
Prompt, please, the main changes which allowed essential to increase the speed of compression of your zopfli version.

I found the interesting image, extent of compression of PNGout is higher than Zopfli.
https://github.com/google/zopfli/iss...ment-168701195

[fhanau]

@lorents:Merging the changes with zopfli is a good idea, but it would be very time-consuming as I changed the API. I also don't know if Google is really interested in improving zopfli(Example: zopfli has the function OptimizeHuffmanForRle which is used to improve the compression of huffman trees. brotli has an improved, but compatible version of that, which is also used in ECT. Why can't they just copy that function?).
As the changes that I've made are significant, making ECT the program with the best deflate compression ratio, maybe I should export my zopfli-based deflate as a library? I think this could also be used in pngwolf.

Mode 4 can be worse than mode 3 as it uses a different initial costmodel state. It perfored better in my tests.

It would be easier to name the things that stayed the same than to name all things that got improved.

I can't guarantee that ECT always has the best compression, as kzip/pngout has a better blocksplitting algorithm.


I have made some progress, especially in the higher modes(Same machine, but using clang as this increases compression):

[Jyrki Alakuijala]

I also don't know if Google is really interested in improving zopfli(Example: zopfli has the function OptimizeHuffmanForRle which is used to improve the compression of huffman trees. brotli has an improved, but compatible version of that, which is also used in ECT.

We are interested in improving zopfli. In brotli the coding of Huffman codes is slightly different, way more efficient than in deflate, so there would likely be some differences in the rle-optimization heuristics, too. I improved the optimize-huffman-for-rle when we were implementing the brotli encoder and just never tried back-porting the changes to zopfli. I'd guess it is a 0.07 % improvement possibility. We'll try to tweak this...

[lorents17]

fhanau, add please a key to ect that PNG could optimize without changes of filters and other parameters. we take paramerta of optimization from the original image.

Jyrki Alakuijala, As I understood you one of the author of zopfli.
I will wait very much for support of OptimizeHuffmanCountsForRle from brotli

[Jaff]

Any chance for ZIP files support?

[fhanau]

Zip support is planned and will be coming soonish. Until then, you can use GZ2ZIP.

[fhanau]

New Version!
Changes:
ZIP Support (limited to one file per archive)
More compression (PNG M1, M8-9)
More speed (PNG M2+)
2 new options:
--allfilters Try all PNG filter strategies
--reuse Do not change PNG filter & strategy

Stats on the PNG images from the Alexa 25 websites, 616 files, ~6MB (OS X 10.10, i52500-S, GCC 5.1):



Binaries will be posted tomorrow.

[Jaff]

A small fix?
Running "ECT -help" will get the message "No compatible files found" instead of help

[lorents17]

I found a serious bug by optimization of this image

C:\Users\Lorents\Desktop\iCatalyst\Tools\apps\ECT.exe --allfilters -2 C:\Users\Lorents\Desktop\1.png

[fhanau]

I found a serious bug by optimization of this image

C:\Users\Lorents\Desktop\iCatalyst\Tools\apps\ECT.exe --allfilters -2 C:\Users\Lorents\Desktop\1.png

A small fix?
Running "ECT -help" will get the message "No compatible files found" instead of help

Thanks, I fixed both bugs.
Edit: With the fix, PNG compression with ECT -7 --allfilters decreased to 17.3981%.

[lorents17]

compile please the new version for Windows
At me at compilation of MSYS2 gives out a mistake

$ make bin
gcc -c -Ofast -std=gnu11 optipng/codec.c optipng/image.c zopfli//util.c zopfli/squeeze.c zopfli/lz77.c zopfli/katajainen.c \
zopfli/blocksplitter.c zopfli/hash.c optipng/opngreduc/opngreduc.c LzFind.c
g++ -Ofast -std=gnu++11 main.cpp blocksplitter.o codec.o hash.o image.o katajainen.o lz77.o opngreduc.o squeeze.o util.o LzFind.o support.cpp zopflipng.cpp zopfli/deflate.cpp zopfli/zopfli_gzip.cpp lodepng/lodepng.cpp lodepng/lodepng_util.cpp optipng/optipng.cpp jpegtran.cpp gztools.cpp mozjpeg/.libs/libjpeg.a libpng/libpng.a zlib/libz.a -o ../ECT
main.cpp: In function 'int main(int, const char**)':
main.cpp:297:5: error: 'vector' is not a member of 'std'
std::vector<int> args;
^
main.cpp:297:17: error: expected primary-expression before 'int'
std::vector<int> args;
^
main.cpp:302:17: error: 'args' was not declared in this scope
args.push_back(i);
^
main.cpp:363:33: error: 'args' was not declared in this scope
PerFileWrapper(argv[args[j]], Options);
^
Makefile:14: ошибка выполнения рецепта для цели «bin»
make: *** [bin] Ошибка 1

[fhanau]

Sorry, I forgot one #include, but that's fixed now.

[Jaff]

Win32 build?

[nikkho]

Win32 build?

Would be nice for me too. Thanks.

[Malloc Voidstar]

0.2.1 compiled for win64/win32. Multi-threading enabled (in official build it isn't)

builds are bdver4 (AMD Excavator), broadwell (AVX), core2 (SSSE3), default (i686 minimal?), native (skylake, AVX2). All but bdver4 work on my systems (x64 haswell-e and skylake) in brief PNG testing, hopefully they work elsewhere. I don't know if choices actually affect speed.

Compiled with GCC 5.3.0. Fixed my compilation so these have JPEG support (my previous builds here did not)

[nikkho]

0.2.1 compiled for win64/win32. Multi-threading enabled (in official build it isn't)

builds are bdver4 (AMD Excavator), broadwell (AVX), core2 (SSSE3), default (i686 minimal?), native (skylake, AVX2). All but bdver4 work on my systems (x64 haswell-e and skylake) in brief PNG testing, hopefully they work elsewhere. I don't know if choices actually affect speed.

Compiled with GCC 5.3.0. Fixed my compilation so these have JPEG support (my previous builds here did not)

Thanks.

[lorents17]

I made the first tests of comparison of iCatalyst and ECT. Results phenomenal.





fhanau, I very much hope to see separately the zopfli project compatible to the original version. I am even better to let out a patch with changes.

[Jaff]

Thank you for ZIP output support but can also add it for input too?

[lorents17]

fhanau,
two versions ECT:

1. ECT 1c483a34d0f46d631d7ddecdac48fc70c418c6cf 05/01/2016
2. ECT 20cd0ed3bfd6fc364bedea8ff698cd34f4a08e8f 06/01/2016

time /c/Users/Lorents/Desktop/ECT-1c483a34d0f46d631d7ddecdac48fc70c418c6cf.exe -M4 /c/Users/Lorents/Desktop/puny_image.png
Processed 1 file
Saved 0B out of 108.85KB (0.0000%)

real    0m9.104s
user    0m0.000s
sys     0m0.000s

time /c/Users/Lorents/Desktop/ECT-20cd0ed3bfd6fc364bedea8ff698cd34f4a08e8f.exe -M4 /c/Users/Lorents/Desktop/puny_image.png
Processed 1 file
Saved 0B out of 108.85KB (0.0000%)

real    0m2.775s
user    0m0.015s
sys     0m0.000s

Prompt how it turned out to increase compression speed so strongly?
It is possible given to apply technology to the original Zopfli version

[dado023]

I made the first tests of comparison of iCatalyst and ECT. Results phenomenal.

fhanau, I very much hope to see separately the zopfli project compatible to the original version. I am even better to let out a patch with changes.

This is very interesting.

But i dont understand how do i use ECT and iCatalyst together to achieve maximum compression of PNG? please explain.

Thank you in advance