Page 14 of 14 FirstFirst ... 4121314
Results 391 to 418 of 418

Thread: ECT, an file optimizer with fast zopfli-like deflate compression

  1. #391
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    873
    Thanks
    49
    Thanked 106 Times in 84 Posts
    Quote Originally Posted by fhanau View Post
    The images, or at least the first few, are not fully opaque. 0.3% of the pixels of the first image are slightly transparent, which means the alpha channel can't be stripped losslessly. The first few images I tested are similar.
    Thank you. Zooming in i can se ther is a slight 1 pixel border where here is some slight transparancy.
    my compare tool is broken then when it comes to alpha

  2. #392
    Member
    Join Date
    Mar 2016
    Location
    USA
    Posts
    49
    Thanks
    7
    Thanked 23 Times in 15 Posts
    Quote Originally Posted by fhanau View Post
    Using an "ultra" value enables aggressive costmodel optimizations in ECT. If ultra is 1, ECT does 1 or more iterations where the costmodel assumes that the actual huffman bit lengths are the rounded theoretical, frequency-based values (usually no rounding takes place). Ultra=2 (or ultra2) does another iteration after all normal/ultra iterations with the ultra approach and also tries to use matches with a longer distance at the same length if this makes sense according to the costmodel. ultra=3 does ultra2 iterations until this does not yield any more saved bits. (Don't worry if you don't fully understand this, it is a really complex topic.)
    I've noticed that ultra2 is quite intensive and ultra3 can literally take months to finish, but doesn't seem to give much if any gain compared to just adding more iterations (at least on the many files I've tested). Is there a better way this functionality can be optimized?

  3. #393
    Member
    Join Date
    Aug 2018
    Location
    India
    Posts
    6
    Thanks
    0
    Thanked 0 Times in 0 Posts

  4. #394
    Member
    Join Date
    Aug 2015
    Location
    Urbana, IL
    Posts
    150
    Thanks
    10
    Thanked 155 Times in 84 Posts
    Quote Originally Posted by MegaByte View Post
    I've noticed that ultra2 is quite intensive and ultra3 can literally take months to finish, but doesn't seem to give much if any gain compared to just adding more iterations (at least on the many files I've tested). Is there a better way this functionality can be optimized?
    If you use ECT for months on one file, you're just wasting CPU time. The ultra2 and ultra3 settings are very inefficient, especially on files with high redundancy. I have added a change that should lead to large speedups for ultra2/3 to the matchfinder-exp branch, but I haven't yet tested it and it needs to be cleaned up. Further improvements are possible by using a more efficient match finder (the main match finder can't be used for ultra2/3 since it needs to find some matches that the main mode does not bother with). Unfortunately, I won't have time to implement new features for ECT in the foreseeable future.

  5. Thanks:

    MegaByte (14th May 2019)

  6. #395
    Member
    Join Date
    Aug 2018
    Location
    India
    Posts
    6
    Thanks
    0
    Thanked 0 Times in 0 Posts

  7. #396
    Member
    Join Date
    Aug 2018
    Location
    India
    Posts
    6
    Thanks
    0
    Thanked 0 Times in 0 Posts

  8. #397
    Member
    Join Date
    Aug 2015
    Location
    Urbana, IL
    Posts
    150
    Thanks
    10
    Thanked 155 Times in 84 Posts
    Quote Originally Posted by shailender_jain View Post
    That tool is lossy, while ECT is only designed for lossless compression for now.

  9. Thanks:

    nikkho (10th April 2019)

  10. #398
    Member
    Join Date
    Mar 2016
    Location
    USA
    Posts
    49
    Thanks
    7
    Thanked 23 Times in 15 Posts
    Quote Originally Posted by fhanau View Post
    I have added a change that should lead to large speedups for ultra2/3 to the matchfinder-exp branch, but I haven't yet tested it and it needs to be cleaned up.
    This branch does help tremendously, but sometimes hits assertions (which corrupt the file if removed).

  11. #399
    Member
    Join Date
    Aug 2015
    Location
    Urbana, IL
    Posts
    150
    Thanks
    10
    Thanked 155 Times in 84 Posts
    Quote Originally Posted by MegaByte View Post
    This branch does help tremendously, but sometimes hits assertions (which corrupt the file if removed).
    Ok, I have opened an issue for this on the GitHub page so I don't forget about it.
    Unfortunately, I am not currently doing any active development on ECT so I don't know when I'll get to fix it.

  12. #400
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    57
    Thanks
    39
    Thanked 25 Times in 16 Posts
    Quote Originally Posted by SnakeSnoke View Post
    @felix: What is the maximum iterations possible? Today I used "ECT -zip -99999" and was able to squeeze out some more bytes.
    After this, I used FileOptimizer (from nikkho), which uses etc, leanify and so on, and the file got 2-4 bytes smaller again. Feels like I could repeat these steps endlessly.


    result: 241 KB (246.939 bytes)
    original zip, 7zip 9.20 ultra: 243 KB (249.049 bytes)
    Here’s what I learned about ZIP compression with ECT:


    1. On PNG files, advpng -i 10000 almost never beats ECT -60500 and if so, only by one or two bytes. That’s obvious because both are built on Zopfli, with ECT being much improved.


    2. I assumed that the same was true for ZIP files, but to my surprise:

    Some Libre Office document: 38,922 B
    7-Zip Ultra ZIP: 12,926 B
    This ZIP file is our start.

    ECT_x64.exe (the Windows version from 2017-08-15) -9 -zip --disable-png --disable-jpg: 12,356 B
    advzip.exe --recompress --shrink-insane: 12,341 B
    Leanify.exe -d 1: 12,337 B
    Something is amiss: ECT directly calls Leanify’s code and should at least reach the same compression! Its Zopfli is also more optimized than advzip’s, so I didn’t expect it to perform worse.
    Checksum on archive content is identical, so no cheating on the original data here.

    This is a very small difference, but it gets larger as iteration counts increase. I have been seeing a 1% improvement on another file last weekend and that got me started in the first place. And it brings us directly to the situation SnakeSnoke described: You can repeat the Leanify-ECT-advzip loop almost endlessly to gain more compression.


    3. I downloaded the current ECT source and compiled it (rough ride because I’m on Windows with Visual Studio). Just like Felix said, ECT’s ZIP optimization is based on Leanify. This in turn calls Zopfli.

    It’ll take some time for me to figure out where it goes wrong, but I just wanted to say:

    Be cautious; ECT is not as good on ZIP as I expected it to be! But it’s probably just a small bug or oversight

  13. Thanks:

    comp1 (21st August 2019)

  14. #401
    Member
    Join Date
    Aug 2015
    Location
    Urbana, IL
    Posts
    150
    Thanks
    10
    Thanked 155 Times in 84 Posts
    Quote Originally Posted by Krishty View Post
    Here’s what I learned about ZIP compression with ECT:


    1. On PNG files, advpng -i 10000 almost never beats ECT -60500 and if so, only by one or two bytes. That’s obvious because both are built on Zopfli, with ECT being much improved.


    2. I assumed that the same was true for ZIP files, but to my surprise:

    Some Libre Office document: 38,922 B
    7-Zip Ultra ZIP: 12,926 B
    This ZIP file is our start.

    ECT_x64.exe (the Windows version from 2017-08-15) -9 -zip --disable-png --disable-jpg: 12,356 B
    advzip.exe --recompress --shrink-insane: 12,341 B
    Leanify.exe -d 1: 12,337 B
    Something is amiss: ECT directly calls Leanify’s code and should at least reach the same compression! Its Zopfli is also more optimized than advzip’s, so I didn’t expect it to perform worse.
    Checksum on archive content is identical, so no cheating on the original data here.

    This is a very small difference, but it gets larger as iteration counts increase. I have been seeing a 1% improvement on another file last weekend and that got me started in the first place. And it brings us directly to the situation SnakeSnoke described: You can repeat the Leanify-ECT-advzip loop almost endlessly to gain more compression.


    3. I downloaded the current ECT source and compiled it (rough ride because I’m on Windows with Visual Studio). Just like Felix said, ECT’s ZIP optimization is based on Leanify. This in turn calls Zopfli.

    It’ll take some time for me to figure out where it goes wrong, but I just wanted to say:

    Be cautious; ECT is not as good on ZIP as I expected it to be! But it’s probably just a small bug or oversight
    Hmm, I'm not sure why this is happening. ECT was initially developed just with speed improvements compared with zopfli in mind and not with maximum compression. It has also only been tested extensively on PNG files, so this might be a case where ECT is not well tuned for the data encountered here. It might make help to examine the zip file and see if block splitting is than in zopfli or if ECT underperforms on small files.

  15. Thanks:

    Krishty (22nd August 2019)

  16. #402
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    57
    Thanks
    39
    Thanked 25 Times in 16 Posts
    @fhanau,

    Which version of ECT is the last good to use? MegaByte said here that the current version(?) hits assertions from time to time … I’m currently using the binaries from przemoc (2018-08-08 ) and from Malloc Voidstar (2017-07-26).

    The reason I’m asking is, there seem to be bugs in ECT that hurt compression (by quite some margin), and I want to know which version I should further test to investigate the issues.

    It seems to me like
    1. all optimization levels from -4 onward lack a certain filter optimization (though this may be fixed with explicit --allfilters)
    2. -7 probably lacks a specific deflate optimization
    3. -10 and onward lack optimizations from -8 and -9


    Details

    I was trying to find the best ECT options for my stuff, and my assumption was that file size shrinks with higher optimization levels, although with dimishing returns. So I expected ECT runs on some image with different options to produce a curve that looks like this:



    It’s not smooth on Y because ECT uses heuristics, and these sometimes work well and sometimes they don’t. It’s also not smooth on X due to errors in measurement.

    So I wrote a program to automatically run tests on ECT (see attached Win32 C++ source), downloaded Lenna from Wikipedia, re-saved it with MSPaint (to undo any prior optimization); recorded -1 to -9 … and to my surprise I got this graph:



    Observe that -1 and -3 perform much better than later optimization levels; not only in terms of size, but also in speed.

    I read this:
    To utilize all the compression options you can also use the "-i" switch where i can be between 1 and 9 to set the mode(higher means better), i >10 for mode 9 and i zopfli iterations and i > 10000 for i/10000 blocksplitting iterations and i%10000 as mode or number of zopfli iterations.
    So naturally the next step was to run 1…9 block splitting passes and up to 92 iterations for Zopfli. This produced the following graph (runs above 10 s omitted for clarity):



    Even with 9 blocksplitting passes and 92 additional Zopfli iterations, ECT never reaches a file as compact with -3!

    I was thinking that maybe Lenna was a bad example, so I picked a vastly different test file: Wikipedia’s pre-2010 logo.

    Same result, if not worse:



    It looks identical with przemoc’s 2018 binaries, which I omit here.

    Now the question is: Is this an issue with filtering or with Zopfli? So I repeated the tests with --reuse (disables filter optimization) and the result matches our expectations a lot better:



    So I assume that there is some filter optimization in -2 and -3 that is not used in -4 and above, and it’s so effective that its missing makes -4 and above practically useless, even at 100x the CPU time.

    We can see another thing in that graph, but it’s even more prominent with Lenna, so I switch to Lenna here. Showing only -1-9 without filtering:



    There is a clear dent at -7, which is worse than -6 and -8 despite -6 using less CPU time. If we just replace -7’s size with -6’s, we almost get a perfect fit to our expected curve:



    Since PNG filters have been disabled, this can only be a problem with the Zopfli call, and only with -7.

    One last thing. Let’s get back to Wikipedia, 1 blocksplit, 40 iterations:



    The two vertices on the far right are -8 and -9. As you can see, the consume a lot of CPU power and provide some improvement, but once we get to -10, ECT optimizes considerably worse and it takes until -30 to catch up. At -40, it surpasses the result of -8 and -9 at much less CPU time.

    These results reproduce with Lenna. While I don’t think this is problematic in itself, it certainly should be mentioned that -10 is often worse than -8 or -9, and that -40 is not only better but also faster than -8 and -9!

    … and it is, above everything else, certainly a bug that all of them perform far worse than -3.

    Next, I will check the effect of the --allfilters option on the result.
    Attached Files Attached Files

  17. Thanks:

    schnaader (6th September 2019)

  18. #403
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    700
    Thanks
    210
    Thanked 267 Times in 157 Posts
    Quote Originally Posted by Krishty View Post
    … and it is, above everything else, certainly a bug that all of them perform far worse than -3.
    Are you able to reproduce the same or similar behaviour in ZopfliPNG? If yes, we can likely fix it there.

  19. #404
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    57
    Thanks
    39
    Thanked 25 Times in 16 Posts
    I’m not sure about the way ECT involves ZopfliPNG. The code seems to be here: https://github.com/fhanau/Efficient-.../main.cpp#L171

    I don’t see any special treatment of modes 3 or 4. It could rather be an issue with OptiPNG (called in line 195) which depends on mode and determines the filter mix for later ZopfliPNG passes.

    While I don’t have any trouble setting up a ZopfliPNG test and waiting a night for the results, I wonder if it would be better to gather a set of representative PNG files and measure performance of individual algorithms instead of entire toolsets like ECT. As in, isolating only the row filtering from ZopfliPNG/OptiPNG/PNGWolf and compare their results. Might pay off more.

  20. #405
    Member
    Join Date
    Aug 2015
    Location
    Urbana, IL
    Posts
    150
    Thanks
    10
    Thanked 155 Times in 84 Posts
    Quote Originally Posted by Krishty View Post
    @fhanau,

    Which version of ECT is the last good to use? MegaByte said here that the current version(?) hits assertions from time to time … I’m currently using the binaries from przemoc (2018-08-08 ) and from Malloc Voidstar (2017-07-26).

    The reason I’m asking is, there seem to be bugs in ECT that hurt compression (by quite some margin), and I want to know which version I should further test to investigate the issues.

    It seems to me like
    1. all optimization levels from -4 onward lack a certain filter optimization (though this may be fixed with explicit --allfilters)
    2. -7 probably lacks a specific deflate optimization
    3. -10 and onward lack optimizations from -8 and -9


    Details

    I was trying to find the best ECT options for my stuff, and my assumption was that file size shrinks with higher optimization levels, although with dimishing returns. So I expected ECT runs on some image with different options to produce a curve that looks like this:



    It’s not smooth on Y because ECT uses heuristics, and these sometimes work well and sometimes they don’t. It’s also not smooth on X due to errors in measurement.

    So I wrote a program to automatically run tests on ECT (see attached Win32 C++ source), downloaded Lenna from Wikipedia, re-saved it with MSPaint (to undo any prior optimization); recorded -1 to -9 … and to my surprise I got this graph:



    Observe that -1 and -3 perform much better than later optimization levels; not only in terms of size, but also in speed.

    I read this:

    So naturally the next step was to run 1…9 block splitting passes and up to 92 iterations for Zopfli. This produced the following graph (runs above 10 s omitted for clarity):



    Even with 9 blocksplitting passes and 92 additional Zopfli iterations, ECT never reaches a file as compact with -3!

    I was thinking that maybe Lenna was a bad example, so I picked a vastly different test file: Wikipedia’s pre-2010 logo.

    Same result, if not worse:



    It looks identical with przemoc’s 2018 binaries, which I omit here.

    Now the question is: Is this an issue with filtering or with Zopfli? So I repeated the tests with --reuse (disables filter optimization) and the result matches our expectations a lot better:



    So I assume that there is some filter optimization in -2 and -3 that is not used in -4 and above, and it’s so effective that its missing makes -4 and above practically useless, even at 100x the CPU time.

    We can see another thing in that graph, but it’s even more prominent with Lenna, so I switch to Lenna here. Showing only -1-9 without filtering:



    There is a clear dent at -7, which is worse than -6 and -8 despite -6 using less CPU time. If we just replace -7’s size with -6’s, we almost get a perfect fit to our expected curve:



    Since PNG filters have been disabled, this can only be a problem with the Zopfli call, and only with -7.

    One last thing. Let’s get back to Wikipedia, 1 blocksplit, 40 iterations:



    The two vertices on the far right are -8 and -9. As you can see, the consume a lot of CPU power and provide some improvement, but once we get to -10, ECT optimizes considerably worse and it takes until -30 to catch up. At -40, it surpasses the result of -8 and -9 at much less CPU time.

    These results reproduce with Lenna. While I don’t think this is problematic in itself, it certainly should be mentioned that -10 is often worse than -8 or -9, and that -40 is not only better but also faster than -8 and -9!

    … and it is, above everything else, certainly a bug that all of them perform far worse than -3.

    Next, I will check the effect of the --allfilters option on the result.
    The version that @MegaByte refered to is on a development branch. The master branch is stable and should not trigger assertions. The assertion causing changes are to the LZ4 and LZMA derived match finders and are unrelated to zopflipng, ECT includes very many changes to zopflipng's deflate and significant changes to lodepng. The tools also have completely different command line interfaces.


    '-10' is not a valid compression setting. The main levels are 1 to 9, other levels are possible as an extension as described in previous comments. The level controls a space-speed tradeoff, so higher modes will naturally be slower. I have noticed that levels above 3 sometimes don't compress better on all files, but they perform better on average.

  21. Thanks:

    Krishty (7th September 2019)

  22. #406
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    57
    Thanks
    39
    Thanked 25 Times in 16 Posts
    @fhanau: Thanks for the clarification! So I’ll use the master branch for further investigation.

    @Jyrki Alakuijala: In my current tests, ZopfliPNG is badly underperforming. I figured it would be clever to test Deflate optimization seperately from filter optimization, so I created test images which only use row filter 0 (none) and their zlib stream is uncompressed. Then I ran pngout, ZopfliPNG and ECT on the files with parameters to not touch line filters (~6000 runs for ECT; ~400 for ZopfliPNG; ~100 for pngout). The result should (to some degree) display Deflate optimization performance:



    As you can see, ZopfliPNG (the blue line) gets easily beaten by pngout as well as ECT. Am I using the wrong parameters (--always-zopflify --filters=p --iterations=XXXX)? Is the executable from https://github.com/imagemin/zopflipn.../zopflipng.exe no good? Test file is attached.

    Best compression results for each tool:
    • 747'937 B after 29.9 s via ect --reuse --strict -30270
    • 748'670 B after 1.14 s via pngout /f6 /force /s0 /n24
    • 748'912 B after 49.9 s via zopflipng --always-zopflify --filters=p --iterations=252

    Edit: I downloaded and compiled the current ZopfliPNG master and called it with the following options:
    Code:
    for(int num_iterations = 1; num_iterations < 999; ++num_iterations) {
        CZopfliPNGOptions options;
        CZopfliPNGSetDefaults(&options);
    
        // Keep it lossless:
        options.lossy_8bit        = 0;
        options.lossy_transparent = 0;
    
        // Disable row filtering:
        auto disableRowFilters = ZopfliPNGFilterStrategy::kStrategyZero;
        options.filter_strategies     = &disableRowFilters;
        options.num_filter_strategies = 1;
        options.auto_filter_strategy  = ZopfliPNGFilterStrategy::kStrategyZero;
    
        options.use_zopfli           = 1;
        options.num_iterations       = num_iterations;
        options.num_iterations_large = num_iterations;
    
        CZopfliPNGOptimize(…);
    Apart from some noise, the result is the same …

    Edit 2: I tried adding advpng to the list because it has 7-Zip compression, but it does not support switching off filters. So I downloaded the source code, made it compatible with VS2019, and called directly into it. Result:
    • the Zopfli optimization basically matches the ZopfliPNG curve in the diagram above; no magic there.
    • 7-Zip’s Deflate is fast but its compression ratio is so bad that it does not even make it into the diagram. It settles at 752'508 B after 0.6 s (17 iterations).
    • advpng sets 7-Zip’s match length limit to 255 instead of 258. Nobody knows why; it does not necessarily improve compression ratio; Igor Pavlov does not remember the details of his implementation.
    So looking into advpng and 7-Zip was pretty useless.

    pngout is closed-source, so there is no chance for me to benchmark anything besides block splitting, which I have already done.

    I looked at OptiPNG as well and it just uses the plain vanilla zlib implementation. No use right now, but it might be interesting for filter benchmarks.

    Next, I’m going to check out Leanify. If this does not add anything to compression ratio, I’ll move on to filter benchmarking.
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	baboon_uncompressed.png 
Views:	12 
Size:	768.6 KB 
ID:	6841  
    Last edited by Krishty; 9th September 2019 at 01:27. Reason: now compiling ZopfliPNG & advpng myself

  23. Thanks:

    dado023 (8th September 2019)

  24. #407
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    57
    Thanks
    39
    Thanked 25 Times in 16 Posts
    In order to get the Deflate benchmarks more fair, I downloaded all compressors I know, compiled them on Windows for x64, and ran them.

    All sample images had row filtering entirely disabled (filter type zero) and were compressed with the Z_STORE setting to avoid bias if tools want to re-use compression choices from original input.

    The tests typically take a day or two, so there’s just a few data points so far: Lenna, Euclid, PNG transparency demonstration; all shown below.





    We’re looking at a very tight size difference here (often just at a per mille of the image). The size differences are really small.

    First, it can definitely be stated that ECT’s Zopfli blows everything else away. For little compression, it’s always several times faster than the Zopfli variantes. For long run times, it constantly achieves higher compression ratios. So high that often the worst run of ECT compresses better than the best run of any Zopfi-related tool.

    But ECT has some weird anomaly where more than 62 iterations where sometimes it becomes incredibly inefficient and suddenly takes ten or thousand(!) times longer to run than 61 or less iterations. This can be seen clearly on Euclid, but it is worse on transparency where I had to omit all runs above 61 iterations because the run-time jumped from twelve seconds to 24,000 (a two-thousand-fold increase)!

    Second, advpng’s 7-Zip seems to be broken. You don’t see it in the benchmarks because it compresses so bad that it didn’t make it into any of the graphs. It’s constantly some percent(!) worse than Zopfli & Co and I just can’t believe that. There has to be a bug in the code, but I couldn’t investigate that yet.

    Now, Zopfli. Advpng made very minor adjustments to its Zopfli (or it is just an outdated version?) and apart from the higher constant overhead, it’s basically the same.

    Leanify’s Zopfli has had some significant changes. It sometimes compresses better, sometimes worse. But on low compression levels, it often compresses better.

    The one problem I see with ECT is that its performance is almost unpredictable. Though better than Zopfli, the difference from -10032 to -10033 can be as large as the difference between Zopfli and ECT. This will be a problem with my upcoming filter benchmarks.

    I should check whether it smoothes when I apply defluff/DeflOpt to the output.

    Input images are attached.
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	euclid.png 
Views:	11 
Size:	286.1 KB 
ID:	6868   Click image for larger version. 

Name:	Lenna.png 
Views:	8 
Size:	768.6 KB 
ID:	6869   Click image for larger version. 

Name:	transparency.jpg 
Views:	9 
Size:	29.5 KB 
ID:	6870  

  25. #408
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    700
    Thanks
    210
    Thanked 267 Times in 157 Posts
    Quote Originally Posted by Krishty View Post
    First, it can definitely be stated that ECT’s Zopfli blows everything else away.
    Do we know why? Better block split heuristics?

    I'd love to see such improvements integrated into the original Zopfli, too.

  26. #409
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    57
    Thanks
    39
    Thanked 25 Times in 16 Posts
    Quote Originally Posted by Jyrki Alakuijala View Post
    Do we know why? Better block split heuristics?

    I'd love to see such improvements integrated into the original Zopfli, too.
    Me as well. Unfortunately, no clue. ECT’s source code is very different, and for example in squeeze.c I see vast floating-point math on symbol costs with comments like:
    /*TODO:Corrections for cost model inaccuracies. There is still much potential here*/
    Sorry, but this is the first time I look into compression code; even plain zlib is still overwhelming to me and ECT looks like a master or doctor thesis to me. Maybe Felix could elaborate on that?

    (Also, I get carried away from the original question – whether ECT’s filtering is better than PNGwolf’s )

  27. #410
    Member
    Join Date
    Mar 2016
    Location
    USA
    Posts
    49
    Thanks
    7
    Thanked 23 Times in 15 Posts
    Quote Originally Posted by Krishty View Post
    (Also, I get carried away from the original question – whether ECT’s filtering is better than PNGwolf’s )
    Some of ECT's filtering code was written by me -- including a genetic algorithm inspired by PNGwolf (as long as you activate it) but with better overall performance especially due to better seeding from the other filter methods. I don't expect PNGwolf to win in any cases currently.
    A number of the other filter algorithms were inspired by Cedric Louvier's post about TruePNG. Since that time, he wrote pingo, which does many of those filters much more efficiently than the brute-force methods included in the ECT code.

  28. Thanks:

    Krishty (16th September 2019)

  29. #411
    Member
    Join Date
    Aug 2015
    Location
    Urbana, IL
    Posts
    150
    Thanks
    10
    Thanked 155 Times in 84 Posts
    Quote Originally Posted by Jyrki Alakuijala View Post
    Do we know why? Better block split heuristics?

    I'd love to see such improvements integrated into the original Zopfli, too.
    I wrote most of ECT years ago, but it mostly comes down to performance improvements in pretty much every part of ECT's deflate, much better caching, a new match finder and better handling of the iterative cost model.

  30. #412
    Member
    Join Date
    Aug 2015
    Location
    Urbana, IL
    Posts
    150
    Thanks
    10
    Thanked 155 Times in 84 Posts
    Quote Originally Posted by Krishty View Post
    Me as well. Unfortunately, no clue. ECT’s source code is very different, and for example in squeeze.c I see vast floating-point math on symbol costs with comments like:
    Sorry, but this is the first time I look into compression code; even plain zlib is still overwhelming to me and ECT looks like a master or doctor thesis to me. Maybe Felix could elaborate on that?

    (Also, I get carried away from the original question – whether ECT’s filtering is better than PNGwolf’s )

    This is a simple heuristic that tunes the LZ cost model based on the results gained from running lazy LZ first when we only have time for one iteration. It is only enabled for PNG when using a low compression level, where it really helps in making ECT with one iteration competitive.

  31. Thanks:

    Krishty (16th September 2019)

  32. #413
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    57
    Thanks
    39
    Thanked 25 Times in 16 Posts
    Quote Originally Posted by MegaByte View Post
    Some of ECT's filtering code was written by me -- including a genetic algorithm inspired by PNGwolf (as long as you activate it) but with better overall performance especially due to better seeding from the other filter methods. I don't expect PNGwolf to win in any cases currently.
    A number of the other filter algorithms were inspired by Cedric Louvier's post about TruePNG. Since that time, he wrote pingo, which does many of those filters much more efficiently than the brute-force methods included in the ECT code.
    Great work, thanks a lot! Guess I’ll do some tests anyway, just out of curiosity

    Quote Originally Posted by fhanau View Post
    This is a simple heuristic that tunes the LZ cost model based on the results gained from running lazy LZ first when we only have time for one iteration. It is only enabled for PNG when using a low compression level, where it really helps in making ECT with one iteration competitive.
    This helps me a lot to get a high-level overview, thanks.

    So – just to establish a check point here – my open questions with ECT are:

    1. Why does -3 perform substantially better than -4 or any higher levels? I know so far:
      • it’s a filter thing (it does not show in my deflate-only benchmarks)
      • a workaround is using the --allfilters option
      • it could be rooted in OptiPNG
    2. How can -61 sometimes take a thousand times longer than -60? (Not -62 vs -61, sorry for the error in my previous post!)
      • definitely a deflate thing; ECT-only
      • could be related to long runs of identical pixels (does not show with Lenna & Co., but with comics and renderings)
    3. How can Leanify & advzip outperform ECT on ZIP files when my benchmarks show such a high superiority of ECT with PNGs?

    I’ll try to find answers in subsequent benchmarks …

  33. #414
    Member
    Join Date
    Aug 2015
    Location
    Urbana, IL
    Posts
    150
    Thanks
    10
    Thanked 155 Times in 84 Posts
    Quote Originally Posted by Krishty View Post
    Great work, thanks a lot! Guess I’ll do some tests anyway, just out of curiosity

    This helps me a lot to get a high-level overview, thanks.

    So – just to establish a check point here – my open questions with ECT are:

    1. Why does -3 perform substantially better than -4 or any higher levels? I know so far:
      • it’s a filter thing (it does not show in my deflate-only benchmarks)
      • a workaround is using the --allfilters option
      • it could be rooted in OptiPNG

    2. How can -61 sometimes take a thousand times longer than -60? (Not -62 vs -61, sorry for the error in my previous post!)
      • definitely a deflate thing; ECT-only
      • could be related to long runs of identical pixels (does not show with Lenna & Co., but with comics and renderings)

    3. How can Leanify & advzip outperform ECT on ZIP files when my benchmarks show such a high superiority of ECT with PNGs?

    I’ll try to find answers in subsequent benchmarks …
    1. -3 does not perform substantially better than -4 in my tests. Have you considered to use a different test set?
    2. -60 and -61 are not supported options. In a future version ECT will reject those arguments so questions like these don't come up anymore.
    3. That depends on the settings used for the tools and the files contained in the zip. ECT was mostly tuned on PNG and text files. On the example you provided, ECT does nineteen bytes worse than leanify, I think occasionally doing that amount worse is acceptable.

  34. #415
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    57
    Thanks
    39
    Thanked 25 Times in 16 Posts
    Quote Originally Posted by fhanau View Post
    1. -3 does not perform substantially better than -4 in my tests. Have you considered to use a different test set?
    Yes, but I didn’t get to the actual tests yet because I wanted to isolate the deflate part first. I’ll let you know once I have the results!

    Quote Originally Posted by fhanau View Post
    2. -60 and -61 are not supported options. In a future version ECT will reject those arguments so questions like these don't come up anymore.
    Sorry if I was unclear – with -60 and -61 I mean -10060/-20060/-30060/etc. It would be a pity to remove those as the fun starts at -xxx11 and the sweet spot for maximal compression seems to be at -xxx30 to -xxx60

    Quote Originally Posted by fhanau View Post
    3. That depends on the settings used for the tools and the files contained in the zip. ECT was mostly tuned on PNG and text files. On the example you provided, ECT does nineteen bytes worse than leanify, I think occasionally doing that amount worse is acceptable.
    Yes, that is absolutely right and it’s absolutely possible that my test set was just bad. However, looking at ECT’s PNG performance – where it is almost never beaten, Leanify being not even close – that could imply some sort of error (if the benchmarks turn out to be valid, again).

    Sorry, I should rather have expressed this as “TODO for me to check out” rather than “questions” … I’m trying not to bother you with guesses here, rather trying to find out what’s going on in my tests and documenting it for others in case it’s useful to them

  35. #416
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    57
    Thanks
    39
    Thanked 25 Times in 16 Posts
    Quote Originally Posted by MegaByte View Post
    Some of ECT's filtering code was written by me -- including a genetic algorithm inspired by PNGwolf (as long as you activate it) but with better overall performance especially due to better seeding from the other filter methods. I don't expect PNGwolf to win in any cases currently. A number of the other filter algorithms were inspired by Cedric Louvier's post about TruePNG. Since that time, he wrote pingo, which does many of those filters much more efficiently than the brute-force methods included in the ECT code.
    I forgot … there is one thing you could help me with. I see that genetic filtering is implemented in lodepng’s encoder, which seems to run after Zopfli. If so, what are the reasons for running it *after* deflate optimization instead of before – wouldn’t that affect compression negatively, especially block splitting?

  36. #417
    Member
    Join Date
    Aug 2015
    Location
    Urbana, IL
    Posts
    150
    Thanks
    10
    Thanked 155 Times in 84 Posts
    Quote Originally Posted by Krishty View Post
    I forgot … there is one thing you could help me with. I see that genetic filtering is implemented in lodepng’s encoder, which seems to run after Zopfli. If so, what are the reasons for running it *after* deflate optimization instead of before – wouldn’t that affect compression negatively, especially block splitting?
    It is not run after ECT's deflate. There is some usage of deflate in the brute force and incremental strategies that may have confused you.

  37. Thanks:

    Krishty (23rd September 2019)

  38. #418
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    57
    Thanks
    39
    Thanked 25 Times in 16 Posts
    Quote Originally Posted by fhanau View Post
    It is not run after ECT's deflate. There is some usage of deflate in the brute force and incremental strategies that may have confused you.
    I checked and it makes sense! Thanks a lot for the clarification

Page 14 of 14 FirstFirst ... 4121314

Similar Threads

  1. defluff - a deflate huffman optimizer
    By jo.henke in forum Data Compression
    Replies: 48
    Last Post: 7th November 2018, 01:04
  2. Intel's fast OSS deflate implementation
    By Bulat Ziganshin in forum Data Compression
    Replies: 16
    Last Post: 23rd May 2016, 18:13
  3. PNG in .ICO file optimizer ?
    By SvenBent in forum Data Compression
    Replies: 9
    Last Post: 21st April 2016, 18:30
  4. deflate, zopfli, lzma intergers og float heavy ?
    By SvenBent in forum The Off-Topic Lounge
    Replies: 2
    Last Post: 23rd September 2015, 16:41
  5. Replies: 3
    Last Post: 14th April 2015, 01:49

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •