Page 2 of 2 FirstFirst 12
Results 31 to 50 of 50

Thread: Huffmix: a PNGOUT -r catalyst

  1. #31
    Member caveman's Avatar
    Join Date
    Jul 2009
    Location
    Strasbourg, France
    Posts
    190
    Thanks
    8
    Thanked 62 Times in 33 Posts
    Expect next version of Huffmix to be 20% faster.
    Prerelease of Huffmix 0.6b2 is available in first post the tarball holds all versions, unfortunately the Windows version in the tarball was not faster, pick the one in the archive labelled huffmix06b2-pre-win.zip to get the 20% speed boost (sorry I didn't check what the compiler had done).
    Last edited by caveman; 6th May 2014 at 12:36.

  2. Thanks:

    Jaff (6th May 2014)

  3. #32
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    873
    Thanks
    49
    Thanked 106 Times in 84 Posts
    cudos. just in time for me to check for time/compression efficiency gains from huffmix compared to straight pngout /r
    Last edited by SvenBent; 6th May 2014 at 05:52.

  4. #33
    Member caveman's Avatar
    Join Date
    Jul 2009
    Location
    Strasbourg, France
    Posts
    190
    Thanks
    8
    Thanked 62 Times in 33 Posts
    I'm afraid you'll have do it again, the Windows binary was effectively a 06b2, but was not faster, please get the new version in the standalone zip archive.
    I've done some benchmarks and this one is effectively about 20% faster.

  5. Thanks:

    Jaff (7th May 2014)

  6. #34
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    873
    Thanks
    49
    Thanked 106 Times in 84 Posts
    when im using huffmix -f it does not copy the smallest file Ehen it errors with "Could not find matching sub-blocks, full stop."

    im having a pngwolf optimized png file.
    i know that pngout /b384 creates a smaller filer

    if i huffmix the pngwolf and the pngout /b384 /f6 /kp file it doesn't output a file
    Last edited by SvenBent; 13th May 2014 at 05:10.

  7. #35
    Member caveman's Avatar
    Join Date
    Jul 2009
    Location
    Strasbourg, France
    Posts
    190
    Thanks
    8
    Thanked 62 Times in 33 Posts
    Quote Originally Posted by SvenBent View Post
    when im using huffmix -f it does not copy the smallest file Ehen it errors with "Could not find matching sub-blocks, full stop."
    Hmm I see, easy to fix, new version soon.

  8. #36
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    873
    Thanks
    49
    Thanked 106 Times in 84 Posts
    Is there any news on a version with the fixed /force option (just copying the smallest file if they are not mixeable)?

  9. #37
    Member Mr_KrzYch00's Avatar
    Join Date
    Apr 2015
    Location
    Poland
    Posts
    65
    Thanks
    10
    Thanked 40 Times in 24 Posts
    Very nice program it can really provide a difference (one PNG file I previously optimized using just pngout and deflopt running a lot of randomized tries reduced further by over 512 bytes).

    However I wish it would support this:
    "Type 0 (uncompressed) block detected, this is not handled by this version."

    The file I tried to huffmix had 7x 0 blocks and 4x 2 blocks (though kzip was forced to run with 5 blocks options hmm). Is it hard to make huffmix copy over areas with 0 type blocks? A lot of EXE compressors like UPX will always result in 0 blocks being created inside GZ/ZIP file...

  10. #38
    Member caveman's Avatar
    Join Date
    Jul 2009
    Location
    Strasbourg, France
    Posts
    190
    Thanks
    8
    Thanked 62 Times in 33 Posts
    Quote Originally Posted by Mr_KrzYch00 View Post
    The file I tried to huffmix had 7x 0 blocks and 4x 2 blocks (though kzip was forced to run with 5 blocks options hmm). Is it hard to make huffmix copy over areas with 0 type blocks? A lot of EXE compressors like UPX will always result in 0 blocks being created inside GZ/ZIP file...
    The problem with 0 type blocks is that the data they carry starts at byte boundaries and this may introduce a few padding bits, this makes their size somewhat variable based on what has been written before. I'm also somewhat annoyed by this situation, I hope to find some time this summer to fix it.
    I have also another tool in the works that may interest you: a zopfli based recompressor that keeps the block boundaries of the entry Deflate stream.

  11. #39
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    873
    Thanks
    49
    Thanked 106 Times in 84 Posts
    Will you look into the " copy smallest file if not mixable" option as well "

    It will really help me multithread some more parts of my png compression batch

  12. #40
    Member Mr_KrzYch00's Avatar
    Join Date
    Apr 2015
    Location
    Poland
    Posts
    65
    Thanks
    10
    Thanked 40 Times in 24 Posts
    Quote Originally Posted by caveman View Post
    The problem with 0 type blocks is that the data they carry starts at byte boundaries and this may introduce a few padding bits, this makes their size somewhat variable based on what has been written before. I'm also somewhat annoyed by this situation, I hope to find some time this summer to fix it.
    I have also another tool in the works that may interest you: a zopfli based recompressor that keeps the block boundaries of the entry Deflate stream.
    I'm now testing advdef using zopfli mode with 99,999 iteraction on some png files. It wins by a small factor over pngout -r in certain cases (mostly only 5-10% of total tests). I noticed with defdb tool that it uses variable block sizes. That's most likely the case how it can win at a times, by better choosing data to compress into blocks, however, I think that Ken's tools use better ways of finding matches, though the problem being that it uses static block split threshold being limited to values every 128. Unfortunately the difference in block splits make it impossible to mix results from both.

    So yeah, a tool that can recompress blocks without altering their boundaries may be a good idea, however, seeing that different block splitting (also being static or variable value) can provide different results, the amount of tries to find best compression ratio is veeery high.

    The tool to somewhat interact with kzip and it's -r option (if possible) as well, to recompress only certain blocks of zopfli produced splitting, might also provide some size reducing, unless finetuning zopfli will provide better results than Ken's solutions OR it actually is better but for kzip block splitting. (can't confirm any of theories)

    Well, it's always interesting to see new tools that can still squeeze these last bytes of those common used ZIP, GZ and PNG files.

    EDIT: Just compiled myself 64-bit version of zopfli and zopflipng. It seems the 2nd can produce smaller files that advdef with brute-force filter (though I'm not sure if that's very safe). Need to test it out a bit...

    EDIT2: it seems zopfli tools win over kzip and pngout. First one I tested on minified javascript files, second one entropy and brute force filer seems to be the most common factor for best size reductions (which pngout fails to reuse), compression mostly is better too. I made KrzYmod for zopfli and zopfliPNG that is available there: http://encode.su/threads/2176-Zopfli...3466#post43466 which allows for finetuning of mentioned by Your post a hardcoded length of 1024 for some length sweet bytes or whatever that's called, can also alter maximum number of blocks to split file to (for which configurable option existed but wasn't exposed to command line interface for changing). The 2nd also provides better ratio on big files. So the best solution would be a tool that can work in both ways, optimize blocks with either kzip -r or zopfli where the file was split to blocks using both tools as well... In other words, to somehow extract a block from deflate file, work on it with any of known best tools (I guess kzip -r and zopfli+iterations?), and put it back into the file while updating all necessary headers, or actually just a tool to extract specified deflate block to some raw file, used to run optimization on by limiting number of blocks to 1 (or maybe it will be possible to further split it to multiple blocks and still put it back into working ZIP file) with various best tools avaiable nowadays (kzip -r and zopfli iterations I guess, maybe let user decide what to use by just giving him some tips on what he CAN'T do in order to block still be valid), and a tool to point that file with, to pack it back as specified deflate block. (not sure if that aproach is actually possible since deflate works on bits not bytes afaik)
    Last edited by Mr_KrzYch00; 27th April 2015 at 17:15. Reason: zopfli tests

  13. #41
    Member Mr_KrzYch00's Avatar
    Join Date
    Apr 2015
    Location
    Poland
    Posts
    65
    Thanks
    10
    Thanked 40 Times in 24 Posts
    Code:
    e:\php\opttools\2>huffmix index.html.gz index2.html.gz index2.html.gza
    index.html.gz is an unknown file type
    For some reason huffmix doesn't like attached file... Also any plans for zip support? I know supporting a lot of files there is a bit of pain since whole central directory data would need to arrayed (or I think uncompressed size+crc32 would provide enough match to find respective offsets in both zips) and then both zip files searched for matches, however zipmix sourcecode may be of help here...
    Attached Files Attached Files
    Last edited by Mr_KrzYch00; 21st May 2015 at 20:42.

  14. #42
    Member just a worm's Avatar
    Join Date
    Aug 2013
    Location
    planet "earth"
    Posts
    96
    Thanks
    29
    Thanked 6 Times in 5 Posts
    @caveman: The compiled files on your website are outdated and the windows version gives a 404.

  15. #43
    Member caveman's Avatar
    Join Date
    Jul 2009
    Location
    Strasbourg, France
    Posts
    190
    Thanks
    8
    Thanked 62 Times in 33 Posts
    Very strange, I'll have a closer look to this file.
    No plan for zip support yet, type 0 blocks and a few other requests first.

  16. #44
    Member caveman's Avatar
    Join Date
    Jul 2009
    Location
    Strasbourg, France
    Posts
    190
    Thanks
    8
    Thanked 62 Times in 33 Posts
    I know, the latest versions are here on encode.su in the first post.

  17. #45
    Member Mr_KrzYch00's Avatar
    Join Date
    Apr 2015
    Location
    Poland
    Posts
    65
    Thanks
    10
    Thanked 40 Times in 24 Posts
    Quote Originally Posted by caveman View Post
    Very strange, I'll have a closer look to this file.
    No plan for zip support yet, type 0 blocks and a few other requests first.
    Actually I know where the problem is. You based huffmix on kzip2gz converter which doesn't include timestamp and extra flags+OS type bytes are zero'ed (that are, in most cases, 2 [compressor used maximum compression - this byte should be either 2 or 4 when CM is 8 - deflate] and 3 [unix OS, 0 means MS-DOS/Win32] respectively). My converter and zopfli KrzYmod puts unix timestamps and exactly 2 and 3 there, since most compressor produce same output. For more information: http://www.onicos.com/staff/iz/formats/gzip.html and https://tools.ietf.org/html/rfc1952 .

    It's not a big deal, I can always use kzip2gz, however, huffmix most likely won't work on normal gz files that aren't first converted to ZIP with my zip2gz then back to GZ with Yours kzip2gz, so kinda painful workaround.

    PS.: Also You could easly support gzips with filename inside (at least properly FSEEK to deflate streams starting position), if offset 3 (byte 4th) is 8 then read file starting at offset 10 (byte 11th) and read until You encounter NULL byte, after this will be deflate stream.

    Part of such code in my gz2zip:
    Code:
      if(flag[0]==8) {
        gzbuffer = (unsigned char*) malloc (1);
        filenamein = (unsigned char*) malloc (1);
        i=0;
        do {
          if(fread(gzbuffer,1,1,gzfile)) {};
          if(gzbuffer[0]!=0) {
            if(realloc(filenamein,i+1) == NULL) {
              fprintf(stderr,"ERROR: Cannot reallocate memory.\n");
              exit(1);
            }
            memcpy(filenamein+i,gzbuffer,1);
            ++i;
          }
        } while(gzbuffer[0]!=0);
    PS2: v13-pre of zopfli KrzYmod has 2 new flags: --cbs (custom block split points) and --cbt (custom block types) so You can easy use it for tests to produce 0 blocks. It's show as example [comparing kzip, zopfli with kzip block splitting and 0 blocks and zopfli + optimizing huffman header] in this post: http://encode.su/threads/2176-Zopfli...ll=1#post43866
    Last edited by Mr_KrzYch00; 26th May 2015 at 18:52.

  18. #46
    Member caveman's Avatar
    Join Date
    Jul 2009
    Location
    Strasbourg, France
    Posts
    190
    Thanks
    8
    Thanked 62 Times in 33 Posts
    Quote Originally Posted by Mr_KrzYch00 View Post
    Actually I know where the problem is. You based huffmix on kzip2gz converter which doesn't include timestamp and extra flags+OS type bytes are zero'ed (that are, in most cases, 2 [compressor used maximum compression - this byte should be either 2 or 4 when CM is 8 - deflate] and 3 [unix OS, 0 means MS-DOS/Win32] respectively). My converter and zopfli KrzYmod puts unix timestamps and exactly 2 and 3 there, since most compressor produce same output. For more information: http://www.onicos.com/staff/iz/formats/gzip.html and https://tools.ietf.org/html/rfc1952 .
    Well not exactly defdb handles this file, huffmix and defdb are sharing the same code for gzip parsing...

  19. #47
    Member Mr_KrzYch00's Avatar
    Join Date
    Apr 2015
    Location
    Poland
    Posts
    65
    Thanks
    10
    Thanked 40 Times in 24 Posts
    The difference between zip2gz and kzip2gz when the file was converted was 6 bytes beginning at offset 3 and ending at offset 8, being 4 bytes of unix timestamp, extra flags byte and OS type byte. Didn't notice anything else.

  20. #48
    Member
    Join Date
    Dec 2015
    Location
    Riga, Latvia
    Posts
    1
    Thanks
    0
    Thanked 1 Time in 1 Post
    I took the liberty to disassemble huffmix to see why it’s choking on gzip files, and I see that when it tries to detect whether the file is gzip as opposed to PNG, it looks for an 8-byte header of exactly 1F8B080000000000, which matches what Mr_KrzYch00 said. Only the first two bytes—or three if you want to report “unknown file type” for hypothetical non-Deflate gzip files—compose the format signature; you shouldn’t expect the rest to be zeros.

  21. Thanks:

    caveman (13th December 2015)

  22. #49
    Member caveman's Avatar
    Join Date
    Jul 2009
    Location
    Strasbourg, France
    Posts
    190
    Thanks
    8
    Thanked 62 Times in 33 Posts
    I'll fix it.

  23. #50
    Member Mr_KrzYch00's Avatar
    Join Date
    Apr 2015
    Location
    Poland
    Posts
    65
    Thanks
    10
    Thanked 40 Times in 24 Posts
    Code:
    cmp     cl, ds:byte_405170[eax]
    jnz     short loc_402101
    inc     eax
    cmp     eax, 8 ; <- change to 4 to reduce iterations to 4 first bytes in GZ files
    jl      short loc_4020B0
    In hex editor with windows binary (0.6b2 prerelease):
    5 314 byte (hex addr: 14C2); change 08 to 04

    This reduces number of loop iterations from 8 to 4 making huffmix not check byte 5-8, in turn fixing timestamp being incorrectly compared with NULL bytes in GZ files. As assembly points fread did read 10 first bytes before the check (meaning the offset is now correctly set at start of deflate stream), so there shouldn't be any corruption afterwards.

  24. Thanks:

    Jaff (28th March 2016)

Page 2 of 2 FirstFirst 12

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •