Page 15 of 15 FirstFirst ... 5131415
Results 421 to 447 of 447

Thread: ECT, an file optimizer with fast zopfli-like deflate compression

  1. #421
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    895
    Thanks
    54
    Thanked 109 Times in 86 Posts
    Quote Originally Posted by fhanau View Post
    You can change it in the source code if it is important to you, but I do not plan to support this in general.
    Thank you. It was mostly for curiosity and testing.
    I am doing a re evaluation of ect against my old pngbest.bat script. trying to figure out which tools are still useable when using ECT

  2. #422
    Member
    Join Date
    Mar 2016
    Location
    Croatia
    Posts
    188
    Thanks
    80
    Thanked 13 Times in 12 Posts
    Quote Originally Posted by SvenBent View Post
    Thank you. It was mostly for curiosity and testing.
    I am doing a re evaluation of ect against my old pngbest.bat script. trying to figure out which tools are still useable when using ECT
    Hi Sven,
    would you be so kind and share your pngbest.bat.
    I am mostly fan of best possible compression, but within in a reasonable compression duration

  3. #423
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    80
    Thanks
    49
    Thanked 39 Times in 24 Posts
    Yes, please share it!

    If anyone is interested, here is how Papa’s Best Optimizer does it:
    1. Copy the file to a temporary location and enforce extension .png
      • because defluff does not work on PNGs without the .png extension
      • also prevents Unicode/max path problems
    2. ECT -30060 --alfilters-b --pal_sort=120, optionally with --strip and --strict
      • 30060 is according to my tests earlier in this thread
    3. defluff
    4. DeflOpt /bk
      • skipped if the file is larger than 512 MiB because it breaks Deflate streams with more than 2³² bits
      • because you said it’s best to run it after defluff
    5. if defluff could not shrink the file and DeflOpt printed Number of files rewritten : 0, optimization stops here; else there has been some gain (even single-bit) and it goes back to 3.
      • This is broken in the current version and will be fixed for the next one; missed a 1-B gain on two out of 200 files.
    The next time I have plenty of spare time, I want to check out Huffmix.

  4. #424
    Member
    Join Date
    Oct 2009
    Location
    usa
    Posts
    59
    Thanks
    1
    Thanked 9 Times in 6 Posts
    Quote Originally Posted by Krishty View Post
    Yes, please share it!
    [*]DeflOpt /bk[LIST][*]skipped if the file is larger than 512 MiB because it breaks Deflate streams with more than 2³² bits
    This deflopt bug has bothered me for years -- decades. Is there any way to patch the binary to overcome this 2^32 bit limit for files? At least making it 2^32 bytes would be help very substantially.

    I take it the source is no longer available or never was. Somehow it must be possible to reverse engineer the binary and eliminate the bits business, change it to bytes...

  5. #425
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    895
    Thanks
    54
    Thanked 109 Times in 86 Posts
    Quote Originally Posted by dado023 View Post
    Hi Sven,
    would you be so kind and share your pngbest.bat.
    I am mostly fan of best possible compression, but within in a reasonable compression duration
    its in the download section of this forum.
    But i am not sure its really optimal now with All the work in ECT. which i why im doing some retesting
    curerlty doing some testing on kodak image suit

  6. #426
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    895
    Thanks
    54
    Thanked 109 Times in 86 Posts
    wow I will admit doing this really show ECT impressiveness

    Code:
    24x 8bit paletted    Bit   Size                           Encoding Time
    Original BMP         32    27.30 MB (28,640,482 bytes)
    Paint save as PNG    32    18.50 MB (19,494,337 bytes)
    PNGout               32    12.30 MB (12,966,158 bytes)    Global Time  =  1010.344 = 00:16:50.344 = 100%
    PNGout /C3            8     9.41 MB ( 9,877,738 bytes)    Global Time  =   122.219 = 00:02:02.219 = 100%
    ECT -9 -strip         8     9.40 MB ( 9,860,903 bytes)    Global Time  =   197.641 = 00:03:17.641 = 100%
    Last edited by SvenBent; 11th December 2019 at 05:01. Reason: Added pngout /c3. Fixed reported size to be filesize rather than diskspace used

  7. #427
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    895
    Thanks
    54
    Thanked 109 Times in 86 Posts
    Does ECT do palette compression /expansion

    1: if a palette contains unused entries does ECT discard the unsued entries ?
    2: if a palette contains the same colors twice does ECT merge them into 1 entry ?
    3: if a palette is only 4 bits ( 16 colors) does ECT try to convert it to 8bits palette for better filtering (as i understand filters works in byte level not pixel level)
    4: Does ECT convert to grey tone if the palette entries all falls withing the colors available in grey tone ?


    Thank you

    -- edit ---

    I guess I could have tested this easily once i got home

  8. #428
    Member
    Join Date
    Aug 2015
    Location
    Urbana, IL
    Posts
    158
    Thanks
    10
    Thanked 162 Times in 90 Posts
    Quote Originally Posted by SvenBent View Post
    Does ECT do palette compression /expansion

    1: if a palette contains unused entries does ECT discard the unsued entries ?
    2: if a palette contains the same colors twice does ECT merge them into 1 entry ?
    3: if a palette is only 4 bits ( 16 colors) does ECT try to convert it to 8bits palette for better filtering (as i understand filters works in byte level not pixel level)
    4: Does ECT convert to grey tone if the palette entries all falls withing the colors available in grey tone ?


    Thank you

    -- edit ---

    I guess I could have tested this easily once i got home
    I think ECT generally rewrites the palette and transforms 1,2 and 4 are done as long as you are using mode 2 or higher.

  9. Thanks:

    SvenBent (11th December 2019)

  10. #429
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    895
    Thanks
    54
    Thanked 109 Times in 86 Posts
    3 days later and im still not done with ect -9 --allfilters-b --pal_sort=120

    i am going to do the full color testing on a faster pc...

  11. #430
    Member
    Join Date
    Aug 2015
    Location
    Urbana, IL
    Posts
    158
    Thanks
    10
    Thanked 162 Times in 90 Posts
    Quote Originally Posted by Krishty View Post
    In order to get the Deflate benchmarks more fair, I downloaded all compressors I know, compiled them on Windows for x64, and ran them.

    But ECT has some weird anomaly where more than 62 iterations where sometimes it becomes incredibly inefficient and suddenly takes ten or thousand(!) times longer to run than 61 or less iterations. This can be seen clearly on Euclid, but it is worse on transparency where I had to omit all runs above 61 iterations because the run-time jumped from twelve seconds to 24,000 (a two-thousand-fold increase)!
    I did some further digging and found out a few things explaining this behaviour:
    1) It looks like ECT does have settings that aren't completely wrong for every compression level you give it, even if it is a setting that is not recommended. However, you should only use options other than -1 to -9 if you know what you are doing and know what the numbers actually stand for (configuring the twice parameter and the iteration count), the values I suggested previously tend to work well while others can have bad space/speed tradeoffs.
    2) Having more than 60 iterations enables the 'ultra2' mode. This has a bad worst-case time complexity and will take awfully long on files with areas with very high redundancy. One of the unstable branches has a possible improvement for this issue and I might integrate it if I have more time for ECT at some point in the future. The transparency image is affected by this issue and settings like -61 should not be used on it. Therefore custom settings like -61 will take a long time, regular -9 works just fine (and is the highest setting recommended for general purpose use).

  12. Thanks:

    Krishty (1st January 2020)

  13. #431
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    80
    Thanks
    49
    Thanked 39 Times in 24 Posts
    Quote Originally Posted by fhanau View Post
    Having more than 60 iterations enables the 'ultra2' mode. This has a bad worst-case time complexity and will take awfully long on files with areas with very high redundancy.
    First, I’m glad to know that it was not an error in my benchmark. Second, it’s very useful information regarding my upcoming ZIP experiments (where I could see *some* slowness with more than 60 iterations, but not nearly as bad with PNG, which can now be explained with ZIPs having less areas of very high redundancy)! Cool!
    One of the unstable branches has a possible improvement for this issue and I might integrate it if I have more time for ECT at some point in the future.
    Even cooler!

  14. #432
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    80
    Thanks
    49
    Thanked 39 Times in 24 Posts
    Weird little request: Could you flush stdout during genetic filtering after printing the current generation? Just a little fflush(stdout) after the printf() in lodepng.cpp?

    Windows has the habit of buffering stdout to batches of 4096 B but only if stdout is not the console. So it looks really good on the console, but once you redirect it to a pipe (e.g. for my Optimizer), it looks frozen until it jumps 2000 generations at a time … the buffer is in the ECT process, so there doesn’t seem a way to do it without touching the code, I’m afraid …

  15. #433
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    895
    Thanks
    54
    Thanked 109 Times in 86 Posts
    Question on pal_sort option

    is there a logical reason for max setting of 120 or is it merly arbitary?
    how does it go about selecting the palette order?

  16. #434
    Member
    Join Date
    Aug 2015
    Location
    Urbana, IL
    Posts
    158
    Thanks
    10
    Thanked 162 Times in 90 Posts
    Quote Originally Posted by SvenBent View Post
    Question on pal_sort option

    is there a logical reason for max setting of 120 or is it merly arbitary?
    how does it go about selecting the palette order?
    There are 120 different ordering strategies in the code implemented by @MegaByte, see https://github.com/fhanau/Efficient-...lodepng.h#L353. The 120 strategies are formed as combinations of 5 ways to define an order of pixels, 2 ways for ascending vs. descending, 3 for transparency handling and 4 for ordering based on color. Refer to the implementation at https://github.com/fhanau/Efficient-...epng.cpp#L1905 for more details.

  17. Thanks (2):

    Krishty (5th January 2020),SvenBent (9th January 2020)

  18. #435
    Member
    Join Date
    Aug 2015
    Location
    Urbana, IL
    Posts
    158
    Thanks
    10
    Thanked 162 Times in 90 Posts
    Quote Originally Posted by Krishty View Post
    Weird little request: Could you flush stdout during genetic filtering after printing the current generation? Just a little fflush(stdout) after the printf() in lodepng.cpp?

    Windows has the habit of buffering stdout to batches of 4096 B but only if stdout is not the console. So it looks really good on the console, but once you redirect it to a pipe (e.g. for my Optimizer), it looks frozen until it jumps 2000 generations at a time … the buffer is in the ECT process, so there doesn’t seem a way to do it without touching the code, I’m afraid …

    Done.

  19. Thanks:

    Krishty (5th January 2020)

  20. #436
    Member
    Join Date
    May 2017
    Location
    Germany
    Posts
    80
    Thanks
    49
    Thanked 39 Times in 24 Posts
    Quote Originally Posted by fhanau View Post
    Having more than 60 iterations enables the 'ultra2' mode. This has a bad worst-case time complexity and will take awfully long on files with areas with very high redundancy. One of the unstable branches has a possible improvement for this issue and I might integrate it if I have more time for ECT at some point in the future.
    Just another confirmation here: I did some quick tests on ZIP and the problem is almost not noticable there. There seems neither a performance cliff after 61 iterations nor stagnation in compression ratio. You say the problems show with areas of high redundancy, and I suspect PNG’s line filters make it have much more redundancy than ordinary files (text/executable) prior to DEFLATE optimization, that’s why it only surfaces there.

  21. #437
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    766
    Thanks
    217
    Thanked 285 Times in 167 Posts
    Quote Originally Posted by SvenBent View Post
    wow I will admit doing this really show ECT impressiveness

    Code:
    24x 8bit paletted    Bit   Size                           Encoding Time
    Original BMP         32    27.30 MB (28,640,482 bytes)
    Paint save as PNG    32    18.50 MB (19,494,337 bytes)
    PNGout               32    12.30 MB (12,966,158 bytes)    Global Time  =  1010.344 = 00:16:50.344 = 100%
    PNGout /C3            8     9.41 MB ( 9,877,738 bytes)    Global Time  =   122.219 = 00:02:02.219 = 100%
    ECT -9 -strip         8     9.40 MB ( 9,860,903 bytes)    Global Time  =   197.641 = 00:03:17.641 = 100%
    Would be more appropriate to compare against ZopfliPNG.

  22. #438
    Member
    Join Date
    Jan 2020
    Location
    West Coast, USA
    Posts
    2
    Thanks
    0
    Thanked 0 Times in 0 Posts
    I have been using OptiPNG, which works fine, but I'm getting too old to see time as free. Do you have ECT binaries for Windows available somewhere?

  23. #439
    Member
    Join Date
    Jun 2018
    Location
    Slovakia
    Posts
    183
    Thanks
    49
    Thanked 13 Times in 13 Posts
    Quote Originally Posted by greenkoopa View Post
    I have been using OptiPNG, which works fine, but I'm getting too old to see time as free. Do you have ECT binaries for Windows available somewhere?
    Windows binary is available in second post.

  24. #440
    Member
    Join Date
    Jan 2020
    Location
    West Coast, USA
    Posts
    2
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by CompressMaster View Post
    Windows binary is available in second post.

    Thank you. I was hoping to benefit from the accomplishments made over the last 4+ years. Is this more of a research project with no official-ish binary releases for end users?

  25. #441
    Member
    Join Date
    Aug 2015
    Location
    Urbana, IL
    Posts
    158
    Thanks
    10
    Thanked 162 Times in 90 Posts
    Quote Originally Posted by greenkoopa View Post
    Thank you. I was hoping to benefit from the accomplishments made over the last 4+ years. Is this more of a research project with no official-ish binary releases for end users?
    There hasn't really been active development lately (college keeps me busy), but I published binaries along with a new release on GitHub. There isn't a whole lot of improvements over previous versions though.

  26. Thanks:

    Krishty (22nd March 2020)

  27. #442
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    895
    Thanks
    54
    Thanked 109 Times in 86 Posts
    I Apologize if this have been answered before. But What is the effect from "--strict "
    Disables unneed edPalet entry removal ?
    Keep meta data?
    Does not remove fully transparant pixels' color data ?


    Thank you

  28. #443
    Member
    Join Date
    Aug 2015
    Location
    Urbana, IL
    Posts
    158
    Thanks
    10
    Thanked 162 Times in 90 Posts
    Quote Originally Posted by SvenBent View Post
    I Apologize if this have been answered before. But What is the effect from "--strict "
    Disables unneed edPalet entry removal ?
    Keep meta data?
    Does not remove fully transparant pixels' color data ?


    Thank you
    --strict disables removing metadata of gzip files and dirty alpha cleaning, operations that could be considered to not be strictly lossless.

  29. #444
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    895
    Thanks
    54
    Thanked 109 Times in 86 Posts
    Quote Originally Posted by fhanau View Post
    --strict disables removing metadata of gzip files and dirty alpha cleaning, operations that could be considered to not be strictly lossless.
    How about unused pallete entries ?
    Im askin cause old time tehchnology for chep "animation" would use pallet cycling so the basic pciture might not reffer to some palle entries but the game will used those palette entries to change the actual pallete reffern making the picture slolwy change color
    like water animations in the old game colonization

    So even though the pallette entriy is not refference to in pixel map. the information is still needed

  30. #445
    Member
    Join Date
    Aug 2015
    Location
    Urbana, IL
    Posts
    158
    Thanks
    10
    Thanked 162 Times in 90 Posts
    Quote Originally Posted by SvenBent View Post
    How about unused pallete entries ?
    Im askin cause old time tehchnology for chep "animation" would use pallet cycling so the basic pciture might not reffer to some palle entries but the game will used those palette entries to change the actual pallete reffern making the picture slolwy change color
    like water animations in the old game colonization

    So even though the pallette entriy is not refference to in pixel map. the information is still needed
    This is not a scenario I was previously aware of, but if I am not mistaken the --reuse flag should preserve the palette (along with color type and filter strategies).

  31. #446
    Member
    Join Date
    Mar 2016
    Location
    USA
    Posts
    50
    Thanks
    7
    Thanked 23 Times in 15 Posts
    Quote Originally Posted by SvenBent View Post
    How about unused pallete entries ?
    Im askin cause old time tehchnology for chep "animation" would use pallet cycling so the basic pciture might not reffer to some palle entries but the game will used those palette entries to change the actual pallete reffern making the picture slolwy change color
    like water animations in the old game colonization

    So even though the pallette entriy is not refference to in pixel map. the information is still needed
    Do you have an example of a real-world product that uses PNG this way?

  32. #447
    Member
    Join Date
    Apr 2013
    Location
    France
    Posts
    14
    Thanks
    0
    Thanked 3 Times in 3 Posts
    ​since ECT does process APNG as PNG, keeping the palette ordering is also critical in this case otherwise it could corrupt fdAT frames

Page 15 of 15 FirstFirst ... 5131415

Similar Threads

  1. defluff - a deflate huffman optimizer
    By jo.henke in forum Data Compression
    Replies: 48
    Last Post: 7th November 2018, 00:04
  2. Intel's fast OSS deflate implementation
    By Bulat Ziganshin in forum Data Compression
    Replies: 16
    Last Post: 23rd May 2016, 17:13
  3. PNG in .ICO file optimizer ?
    By SvenBent in forum Data Compression
    Replies: 9
    Last Post: 21st April 2016, 17:30
  4. deflate, zopfli, lzma intergers og float heavy ?
    By SvenBent in forum The Off-Topic Lounge
    Replies: 2
    Last Post: 23rd September 2015, 15:41
  5. Replies: 3
    Last Post: 14th April 2015, 00:49

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •