Page 2 of 15 FirstFirst 123412 ... LastLast
Results 31 to 60 of 447

Thread: ECT, an file optimizer with fast zopfli-like deflate compression

  1. #31
    Member nikkho's Avatar
    Join Date
    Jul 2011
    Location
    Spain
    Posts
    550
    Thanks
    222
    Thanked 165 Times in 106 Posts
    Quote Originally Posted by fhanau View Post
    Fixed.
    Thanks for the x64 build.

  2. #32
    Member przemoc's Avatar
    Join Date
    Aug 2011
    Location
    Poland
    Posts
    44
    Thanks
    3
    Thanked 23 Times in 13 Posts
    I've built it on my W7 x64 in up-to-date MSYS2's MinGW-w64 Win64 Shell using recent:
    Code:
    gcc version 5.3.0 (Rev1, Built by MSYS2 project)
    after performing
    Code:
    ( cd src/mozjpeg/ && autoreconf -fi )
    adding LDFLAGS to src/Makefile:
    Code:
    LDFLAGS= -static -static-libgcc -static-libstdc++
    and using these LDFLAGS during linking.
    Binary in the attachment is stripped.

    Seems to be working fine with recent fix.

    Felix, don't you think using
    #if defined(...) && !defined(...) && ...

    would be a bit nicer than 3 levels of #ifdef hell?
    Attached Files Attached Files

  3. #33
    Member przemoc's Avatar
    Join Date
    Aug 2011
    Location
    Poland
    Posts
    44
    Thanks
    3
    Thanked 23 Times in 13 Posts
    New build using latest commit (with fixed Makefile).

    Felix, you forgot to update Makefile after removing zlibWrapper.
    Attached Files Attached Files

  4. #34
    Member
    Join Date
    Apr 2011
    Location
    Russia
    Posts
    168
    Thanks
    163
    Thanked 9 Times in 8 Posts
    I made two test, here results:

    test 1 (8600 PNG):
    iCatalyst - Advanced - 604 846 779 byte - 0:42:13
    ect from 25/12/2015 - M2 - 604 383 889 byte - 0:37:03
    ect from 01/01/2015 - M2 - 605 726 841 byte - 0:34:14

    test 2 (3800 PNG):
    iCatalyst - Advanced - 1 184 876 290 byte - 0:47:16
    ect from 25/12/2015 - M2 - 1 181 220 466 byte - 0:42:46
    ect from 01/01/2015 - M2 - 1 184 451 798 byte - 0:38:42
    Last edited by lorents17; 1st January 2016 at 16:55.

  5. #35
    Member
    Join Date
    Aug 2015
    Location
    Urbana, IL
    Posts
    158
    Thanks
    10
    Thanked 162 Times in 90 Posts
    Quote Originally Posted by lorents17 View Post
    I made two test, here results:

    test 1 (8600 PNG):
    iCatalyst - Advanced - 604 846 779 byte - 0:42:13
    ect from 25/12/2015 - M2 - 604 383 889 byte - 0:37:03
    ect from 01/01/2015 - M2 - 605 726 841 byte - 0:34:14

    test 2 (3800 PNG):
    iCatalyst - Advanced - 1 184 876 290 byte - 0:47:16
    ect from 25/12/2015 - M2 - 1 181 220 466 byte - 0:42:46
    ect from 01/01/2015 - M2 - 1 184 451 798 byte - 0:38:42
    This happened probably due to this and this. These commits improve speed and text compression, but reduce PNG compression as blocksplitting and costmodel get worse. But I also made improvements since 25/12 and the compression on my own test images actually got better.

  6. #36
    Member
    Join Date
    Apr 2011
    Location
    Russia
    Posts
    168
    Thanks
    163
    Thanked 9 Times in 8 Posts
    New test:

    test 1 (8600 PNG):
    iCatalyst - Xtreme - 598 100 342 byte - 2:42:52
    ect by 25/12/2015 - M4 - 598 402 908 byte - 2:49:57
    ect by 01/01/2015 - M4 - 599 312 198 byte - 2:33:20

    test 2 (3800 PNG):
    iCatalyst - Xtreme - 1 170 480 787 byte - 3:52:29
    ect by 25/12/2015 - M4 - 1 170 850 232 byte - 4:48:48
    ect by 01/01/2015 - M4 - 1 172 646 702 byte - 4:23:33
    Last edited by lorents17; 8th February 2016 at 11:22.

  7. #37
    Member
    Join Date
    Apr 2011
    Location
    Russia
    Posts
    168
    Thanks
    163
    Thanked 9 Times in 8 Posts
    test 1 (8600 PNG):
    ect by 06/01/2015 - M2 - 604 801 930 byte - 0:34:22
    ect by 06/01/2015 - M4 - 599 303 054 byte - 1:05:31

    test 2 (3800 PNG):
    ect by 06/01/2015 - M2 - 1 182 829 456 byte - 0:39:41
    ect by 06/01/2015 - M4 - 1 172 638 358 byte - 1:30:58

    Results of the last dough simply impress, especially the mode - M4
    Now I test the new version.

  8. #38
    Member
    Join Date
    Aug 2015
    Location
    Urbana, IL
    Posts
    158
    Thanks
    10
    Thanked 162 Times in 90 Posts
    I released Version 0.2.
    Changes:
    Nine modes, usage is now -1 to -9 instead of -M1 to -M5
    Much higher speed when using many iterations as I added an matches cache. lorents, this caused your speed improvements.

    New stats on enwik8 (gcc 5.1, Intel i5-2500S @ 2.7Ghz, OS X 10.10):
    Click image for larger version. 

Name:	Bildschirmfoto 2016-01-08 um 13.19.54.png 
Views:	245 
Size:	8.8 KB 
ID:	4019
    Last edited by fhanau; 8th January 2016 at 14:21. Reason: Fix table

  9. Thanks (3):

    AiZ (7th January 2016),lorents17 (8th January 2016),nikkho (8th January 2016)

  10. #39
    Member
    Join Date
    Apr 2011
    Location
    Russia
    Posts
    168
    Thanks
    163
    Thanked 9 Times in 8 Posts
    fhanau
    I very much ask you, make if it is possible, the zopfli version compatible to the original version.
    And it is even better to make patches, for the original zopfli version and to send to developers.

  11. #40
    Member nikkho's Avatar
    Join Date
    Jul 2011
    Location
    Spain
    Posts
    550
    Thanks
    222
    Thanked 165 Times in 106 Posts
    Thanks. Could you post on the releases page, also the win32 one?

  12. #41
    Member
    Join Date
    Apr 2011
    Location
    Russia
    Posts
    168
    Thanks
    163
    Thanked 9 Times in 8 Posts
    Code:
    test 1 (8600 PNG):
    ect -2    607 423 500    0:35:23
    ect -3    604 707 780    0:38:49
    ect -4    605 112 805    0:39:18
    ect -5    602 818 906    0:44:29
    ect -6    600 773 213    0:51:02
    ect -7    599 199 736    1:01:22
    ect -8    597 995 893    2:12:37
    ect -9    597 985 702    2:23:06
    Code:
    test 1 (3800 PNG):
    ect -2    1 190 882 178    0:34:27
    ect -3    1 182 829 456    0:43:23
    ect -4    1 184 709 350    0:46:37
    ect -5    1 180 032 491    0:55:41
    ect -6    1 175 521 282    1:06:52
    ect -7    1 172 470 148    1:31:28
    ect -8    1 170 570 598    3:29:24
    ect -9    1 170 570 598    3:42:33
    It is very interesting to test your zopfli version with a combination of TruePNG and PNGWofl. It is sure results of compression it is possible to increase significantly.
    Prompt, please, the main changes which allowed essential to increase the speed of compression of your zopfli version.

    I found the interesting image, extent of compression of PNGout is higher than Zopfli.
    https://github.com/google/zopfli/iss...ment-168701195
    Last edited by lorents17; 10th January 2016 at 10:08.

  13. #42
    Member
    Join Date
    Aug 2015
    Location
    Urbana, IL
    Posts
    158
    Thanks
    10
    Thanked 162 Times in 90 Posts
    @lorents:Merging the changes with zopfli is a good idea, but it would be very time-consuming as I changed the API. I also don't know if Google is really interested in improving zopfli(Example: zopfli has the function OptimizeHuffmanForRle which is used to improve the compression of huffman trees. brotli has an improved, but compatible version of that, which is also used in ECT. Why can't they just copy that function?).
    As the changes that I've made are significant, making ECT the program with the best deflate compression ratio, maybe I should export my zopfli-based deflate as a library? I think this could also be used in pngwolf.

    Mode 4 can be worse than mode 3 as it uses a different initial costmodel state. It perfored better in my tests.

    It would be easier to name the things that stayed the same than to name all things that got improved.

    I can't guarantee that ECT always has the best compression, as kzip/pngout has a better blocksplitting algorithm.


    I have made some progress, especially in the higher modes(Same machine, but using clang as this increases compression):
    Click image for larger version. 

Name:	pic.png 
Views:	248 
Size:	4.4 KB 
ID:	4050

  14. #43
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    766
    Thanks
    217
    Thanked 285 Times in 167 Posts
    Quote Originally Posted by fhanau View Post
    I also don't know if Google is really interested in improving zopfli(Example: zopfli has the function OptimizeHuffmanForRle which is used to improve the compression of huffman trees. brotli has an improved, but compatible version of that, which is also used in ECT.
    We are interested in improving zopfli. In brotli the coding of Huffman codes is slightly different, way more efficient than in deflate, so there would likely be some differences in the rle-optimization heuristics, too. I improved the optimize-huffman-for-rle when we were implementing the brotli encoder and just never tried back-porting the changes to zopfli. I'd guess it is a 0.07 % improvement possibility. We'll try to tweak this...
    Last edited by Jyrki Alakuijala; 31st January 2016 at 02:55.

  15. #44
    Member
    Join Date
    Apr 2011
    Location
    Russia
    Posts
    168
    Thanks
    163
    Thanked 9 Times in 8 Posts
    fhanau, add please a key to ect that PNG could optimize without changes of filters and other parameters. we take paramerta of optimization from the original image.

    Jyrki Alakuijala, As I understood you one of the author of zopfli.
    I will wait very much for support of OptimizeHuffmanCountsForRle from brotli
    Last edited by lorents17; 31st January 2016 at 12:55.

  16. #45
    Member Jaff's Avatar
    Join Date
    Oct 2012
    Location
    Dracula's country
    Posts
    104
    Thanks
    115
    Thanked 22 Times in 18 Posts
    Any chance for ZIP files support?

  17. #46
    Member
    Join Date
    Aug 2015
    Location
    Urbana, IL
    Posts
    158
    Thanks
    10
    Thanked 162 Times in 90 Posts
    Zip support is planned and will be coming soonish. Until then, you can use GZ2ZIP.

  18. Thanks:

    Jaff (3rd February 2016)

  19. #47
    Member
    Join Date
    Aug 2015
    Location
    Urbana, IL
    Posts
    158
    Thanks
    10
    Thanked 162 Times in 90 Posts
    New Version!
    Changes:
    ZIP Support (limited to one file per archive)
    More compression (PNG M1, M8-9)
    More speed (PNG M2+)
    2 new options:
    --allfilters Try all PNG filter strategies
    --reuse Do not change PNG filter & strategy

    Stats on the PNG images from the Alexa 25 websites, 616 files, ~6MB (OS X 10.10, i52500-S, GCC 5.1):
    Click image for larger version. 

Name:	Bildschirmfoto 2016-02-07 um 19.52.06.png 
Views:	273 
Size:	61.3 KB 
ID:	4069


    Binaries will be posted tomorrow.
    Last edited by fhanau; 7th February 2016 at 20:57.

  20. Thanks (3):

    Jaff (8th February 2016),Jyrki Alakuijala (7th February 2016),lorents17 (7th February 2016)

  21. #48
    Member Jaff's Avatar
    Join Date
    Oct 2012
    Location
    Dracula's country
    Posts
    104
    Thanks
    115
    Thanked 22 Times in 18 Posts
    A small fix?
    Running "ECT -help" will get the message "No compatible files found" instead of help

  22. #49
    Member
    Join Date
    Apr 2011
    Location
    Russia
    Posts
    168
    Thanks
    163
    Thanked 9 Times in 8 Posts
    I found a serious bug by optimization of this image
    Code:
    C:\Users\Lorents\Desktop\iCatalyst\Tools\apps\ECT.exe --allfilters -2 C:\Users\Lorents\Desktop\1.png
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	1.png 
Views:	167 
Size:	48.7 KB 
ID:	4070   Click image for larger version. 

Name:	2.png 
Views:	190 
Size:	7.7 KB 
ID:	4071  

  23. #50
    Member
    Join Date
    Aug 2015
    Location
    Urbana, IL
    Posts
    158
    Thanks
    10
    Thanked 162 Times in 90 Posts
    Quote Originally Posted by lorents17 View Post
    I found a serious bug by optimization of this image
    Code:
    C:\Users\Lorents\Desktop\iCatalyst\Tools\apps\ECT.exe --allfilters -2 C:\Users\Lorents\Desktop\1.png
    Quote Originally Posted by Jaff View Post
    A small fix?
    Running "ECT -help" will get the message "No compatible files found" instead of help
    Thanks, I fixed both bugs.
    Edit: With the fix, PNG compression with ECT -7 --allfilters decreased to 17.3981%.
    Last edited by fhanau; 8th February 2016 at 14:54.

  24. Thanks:

    lorents17 (8th February 2016)

  25. #51
    Member
    Join Date
    Apr 2011
    Location
    Russia
    Posts
    168
    Thanks
    163
    Thanked 9 Times in 8 Posts
    compile please the new version for Windows
    At me at compilation of MSYS2 gives out a mistake

    $ make bin
    gcc -c -Ofast -std=gnu11 optipng/codec.c optipng/image.c zopfli//util.c zopfli/squeeze.c zopfli/lz77.c zopfli/katajainen.c \
    zopfli/blocksplitter.c zopfli/hash.c optipng/opngreduc/opngreduc.c LzFind.c
    g++ -Ofast -std=gnu++11 main.cpp blocksplitter.o codec.o hash.o image.o katajainen.o lz77.o opngreduc.o squeeze.o util.o LzFind.o support.cpp zopflipng.cpp zopfli/deflate.cpp zopfli/zopfli_gzip.cpp lodepng/lodepng.cpp lodepng/lodepng_util.cpp optipng/optipng.cpp jpegtran.cpp gztools.cpp mozjpeg/.libs/libjpeg.a libpng/libpng.a zlib/libz.a -o ../ECT
    main.cpp: In function 'int main(int, const char**)':
    main.cpp:297:5: error: 'vector' is not a member of 'std'
    std::vector<int> args;
    ^
    main.cpp:297:17: error: expected primary-expression before 'int'
    std::vector<int> args;
    ^
    main.cpp:302:17: error: 'args' was not declared in this scope
    args.push_back(i);
    ^
    main.cpp:363:33: error: 'args' was not declared in this scope
    PerFileWrapper(argv[args[j]], Options);
    ^
    Makefile:14: ошибка выполнения рецепта для цели «bin»
    make: *** [bin] Ошибка 1

  26. #52
    Member
    Join Date
    Aug 2015
    Location
    Urbana, IL
    Posts
    158
    Thanks
    10
    Thanked 162 Times in 90 Posts
    Sorry, I forgot one #include, but that's fixed now.
    Attached Files Attached Files

  27. Thanks (2):

    lorents17 (8th February 2016),Malloc Voidstar (8th February 2016)

  28. #53
    Member Jaff's Avatar
    Join Date
    Oct 2012
    Location
    Dracula's country
    Posts
    104
    Thanks
    115
    Thanked 22 Times in 18 Posts
    Win32 build?

  29. #54
    Member nikkho's Avatar
    Join Date
    Jul 2011
    Location
    Spain
    Posts
    550
    Thanks
    222
    Thanked 165 Times in 106 Posts
    Quote Originally Posted by Jaff View Post
    Win32 build?
    Would be nice for me too. Thanks.

  30. #55
    Member
    Join Date
    Feb 2016
    Location
    USA
    Posts
    41
    Thanks
    27
    Thanked 63 Times in 24 Posts
    0.2.1 compiled for win64/win32. Multi-threading enabled (in official build it isn't)

    builds are bdver4 (AMD Excavator), broadwell (AVX), core2 (SSSE3), default (i686 minimal?), native (skylake, AVX2). All but bdver4 work on my systems (x64 haswell-e and skylake) in brief PNG testing, hopefully they work elsewhere. I don't know if choices actually affect speed.

    Compiled with GCC 5.3.0. Fixed my compilation so these have JPEG support (my previous builds here did not)
    Attached Files Attached Files
    Last edited by Malloc Voidstar; 10th February 2016 at 02:27.

  31. Thanks (2):

    Jaff (11th February 2016),nikkho (10th February 2016)

  32. #56
    Member nikkho's Avatar
    Join Date
    Jul 2011
    Location
    Spain
    Posts
    550
    Thanks
    222
    Thanked 165 Times in 106 Posts
    Quote Originally Posted by Malloc Voidstar View Post
    0.2.1 compiled for win64/win32. Multi-threading enabled (in official build it isn't)

    builds are bdver4 (AMD Excavator), broadwell (AVX), core2 (SSSE3), default (i686 minimal?), native (skylake, AVX2). All but bdver4 work on my systems (x64 haswell-e and skylake) in brief PNG testing, hopefully they work elsewhere. I don't know if choices actually affect speed.

    Compiled with GCC 5.3.0. Fixed my compilation so these have JPEG support (my previous builds here did not)
    Thanks.

  33. #57
    Member
    Join Date
    Apr 2011
    Location
    Russia
    Posts
    168
    Thanks
    163
    Thanked 9 Times in 8 Posts
    I made the first tests of comparison of iCatalyst and ECT. Results phenomenal.

    Click image for larger version. 

Name:	1.PNG 
Views:	239 
Size:	11.6 KB 
ID:	4087

    Click image for larger version. 

Name:	2.PNG 
Views:	208 
Size:	11.0 KB 
ID:	4088

    fhanau, I very much hope to see separately the zopfli project compatible to the original version. I am even better to let out a patch with changes.
    Last edited by lorents17; 11th February 2016 at 21:57.

  34. #58
    Member Jaff's Avatar
    Join Date
    Oct 2012
    Location
    Dracula's country
    Posts
    104
    Thanks
    115
    Thanked 22 Times in 18 Posts
    Thank you for ZIP output support but can also add it for input too?

  35. #59
    Member
    Join Date
    Apr 2011
    Location
    Russia
    Posts
    168
    Thanks
    163
    Thanked 9 Times in 8 Posts
    fhanau,
    two versions ECT:

    1. ECT 1c483a34d0f46d631d7ddecdac48fc70c418c6cf 05/01/2016
    2. ECT 20cd0ed3bfd6fc364bedea8ff698cd34f4a08e8f 06/01/2016

    Code:
    time /c/Users/Lorents/Desktop/ECT-1c483a34d0f46d631d7ddecdac48fc70c418c6cf.exe -M4 /c/Users/Lorents/Desktop/puny_image.png
    Processed 1 file
    Saved 0B out of 108.85KB (0.0000%)
    
    real    0m9.104s
    user    0m0.000s
    sys     0m0.000s
    
    time /c/Users/Lorents/Desktop/ECT-20cd0ed3bfd6fc364bedea8ff698cd34f4a08e8f.exe -M4 /c/Users/Lorents/Desktop/puny_image.png
    Processed 1 file
    Saved 0B out of 108.85KB (0.0000%)
    
    real    0m2.775s
    user    0m0.015s
    sys     0m0.000s
    Prompt how it turned out to increase compression speed so strongly?
    It is possible given to apply technology to the original Zopfli version
    Last edited by lorents17; 23rd February 2016 at 13:09.

  36. #60
    Member
    Join Date
    Mar 2016
    Location
    Croatia
    Posts
    188
    Thanks
    80
    Thanked 13 Times in 12 Posts
    Quote Originally Posted by lorents17 View Post
    I made the first tests of comparison of iCatalyst and ECT. Results phenomenal.
    Quote Originally Posted by lorents17 View Post

    Click image for larger version. 

Name:	1.PNG 
Views:	239 
Size:	11.6 KB 
ID:	4087

    Click image for larger version. 

Name:	2.PNG 
Views:	208 
Size:	11.0 KB 
ID:	4088

    fhanau, I very much hope to see separately the zopfli project compatible to the original version. I am even better to let out a patch with changes.
    This is very interesting.

    But i dont understand how do i use ECT and iCatalyst together to achieve maximum compression of PNG? please explain.

    Thank you in advance
    Last edited by dado023; 2nd March 2016 at 00:17.

Page 2 of 15 FirstFirst 123412 ... LastLast

Similar Threads

  1. defluff - a deflate huffman optimizer
    By jo.henke in forum Data Compression
    Replies: 48
    Last Post: 7th November 2018, 00:04
  2. Intel's fast OSS deflate implementation
    By Bulat Ziganshin in forum Data Compression
    Replies: 16
    Last Post: 23rd May 2016, 17:13
  3. PNG in .ICO file optimizer ?
    By SvenBent in forum Data Compression
    Replies: 9
    Last Post: 21st April 2016, 17:30
  4. deflate, zopfli, lzma intergers og float heavy ?
    By SvenBent in forum The Off-Topic Lounge
    Replies: 2
    Last Post: 23rd September 2015, 15:41
  5. Replies: 3
    Last Post: 14th April 2015, 00:49

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •