Huffmix: a PNGOUT -r catalyst

[caveman]

Expect next version of Huffmix to be 20% faster.
Prerelease of Huffmix 0.6b2 is available in first post the tarball holds all versions, unfortunately the Windows version in the tarball was not faster, pick the one in the archive labelled huffmix06b2-pre-win.zip to get the 20% speed boost (sorry I didn't check what the compiler had done).

[SvenBent]

cudos. just in time for me to check for time/compression efficiency gains from huffmix compared to straight pngout /r

[caveman]

I'm afraid you'll have do it again, the Windows binary was effectively a 06b2, but was not faster, please get the new version in the standalone zip archive.
I've done some benchmarks and this one is effectively about 20% faster.

[SvenBent]

when im using huffmix -f it does not copy the smallest file Ehen it errors with "Could not find matching sub-blocks, full stop."

im having a pngwolf optimized png file.
i know that pngout /b384 creates a smaller filer

if i huffmix the pngwolf and the pngout /b384 /f6 /kp file it doesn't output a file

[caveman]

when im using huffmix -f it does not copy the smallest file Ehen it errors with "Could not find matching sub-blocks, full stop."

Hmm I see, easy to fix, new version soon.

[SvenBent]

Is there any news on a version with the fixed /force option (just copying the smallest file if they are not mixeable)?

[Mr_KrzYch00]

Very nice program it can really provide a difference (one PNG file I previously optimized using just pngout and deflopt running a lot of randomized tries reduced further by over 512 bytes).

However I wish it would support this:
"Type 0 (uncompressed) block detected, this is not handled by this version."

The file I tried to huffmix had 7x 0 blocks and 4x 2 blocks (though kzip was forced to run with 5 blocks options hmm). Is it hard to make huffmix copy over areas with 0 type blocks? A lot of EXE compressors like UPX will always result in 0 blocks being created inside GZ/ZIP file...

[caveman]

The file I tried to huffmix had 7x 0 blocks and 4x 2 blocks (though kzip was forced to run with 5 blocks options hmm). Is it hard to make huffmix copy over areas with 0 type blocks? A lot of EXE compressors like UPX will always result in 0 blocks being created inside GZ/ZIP file...

The problem with 0 type blocks is that the data they carry starts at byte boundaries and this may introduce a few padding bits, this makes their size somewhat variable based on what has been written before. I'm also somewhat annoyed by this situation, I hope to find some time this summer to fix it.
I have also another tool in the works that may interest you: a zopfli based recompressor that keeps the block boundaries of the entry Deflate stream.

[SvenBent]

Will you look into the " copy smallest file if not mixable" option as well "

It will really help me multithread some more parts of my png compression batch

[Mr_KrzYch00]

The problem with 0 type blocks is that the data they carry starts at byte boundaries and this may introduce a few padding bits, this makes their size somewhat variable based on what has been written before. I'm also somewhat annoyed by this situation, I hope to find some time this summer to fix it.
I have also another tool in the works that may interest you: a zopfli based recompressor that keeps the block boundaries of the entry Deflate stream.

I'm now testing advdef using zopfli mode with 99,999 iteraction on some png files. It wins by a small factor over pngout -r in certain cases (mostly only 5-10% of total tests). I noticed with defdb tool that it uses variable block sizes. That's most likely the case how it can win at a times, by better choosing data to compress into blocks, however, I think that Ken's tools use better ways of finding matches, though the problem being that it uses static block split threshold being limited to values every 128. Unfortunately the difference in block splits make it impossible to mix results from both.

So yeah, a tool that can recompress blocks without altering their boundaries may be a good idea, however, seeing that different block splitting (also being static or variable value) can provide different results, the amount of tries to find best compression ratio is veeery high.

The tool to somewhat interact with kzip and it's -r option (if possible) as well, to recompress only certain blocks of zopfli produced splitting, might also provide some size reducing, unless finetuning zopfli will provide better results than Ken's solutions OR it actually is better but for kzip block splitting. (can't confirm any of theories)

Well, it's always interesting to see new tools that can still squeeze these last bytes of those common used ZIP, GZ and PNG files.

EDIT: Just compiled myself 64-bit version of zopfli and zopflipng. It seems the 2nd can produce smaller files that advdef with brute-force filter (though I'm not sure if that's very safe). Need to test it out a bit...

EDIT2: it seems zopfli tools win over kzip and pngout. First one I tested on minified javascript files, second one entropy and brute force filer seems to be the most common factor for best size reductions (which pngout fails to reuse), compression mostly is better too. I made KrzYmod for zopfli and zopfliPNG that is available there: http://encode.su/threads/2176-Zopfli...3466#post43466 which allows for finetuning of mentioned by Your post a hardcoded length of 1024 for some length sweet bytes or whatever that's called, can also alter maximum number of blocks to split file to (for which configurable option existed but wasn't exposed to command line interface for changing). The 2nd also provides better ratio on big files. So the best solution would be a tool that can work in both ways, optimize blocks with either kzip -r or zopfli where the file was split to blocks using both tools as well... In other words, to somehow extract a block from deflate file, work on it with any of known best tools (I guess kzip -r and zopfli+iterations?), and put it back into the file while updating all necessary headers, or actually just a tool to extract specified deflate block to some raw file, used to run optimization on by limiting number of blocks to 1 (or maybe it will be possible to further split it to multiple blocks and still put it back into working ZIP file) with various best tools avaiable nowadays (kzip -r and zopfli iterations I guess, maybe let user decide what to use by just giving him some tips on what he CAN'T do in order to block still be valid), and a tool to point that file with, to pack it back as specified deflate block. (not sure if that aproach is actually possible since deflate works on bits not bytes afaik)

[Mr_KrzYch00]

e:\php\opttools\2>huffmix index.html.gz index2.html.gz index2.html.gza
index.html.gz is an unknown file type

For some reason huffmix doesn't like attached file... Also any plans for zip support? I know supporting a lot of files there is a bit of pain since whole central directory data would need to arrayed (or I think uncompressed size+crc32 would provide enough match to find respective offsets in both zips) and then both zip files searched for matches, however zipmix sourcecode may be of help here...

[just a worm]

@caveman: The compiled files on your website are outdated and the windows version gives a 404.

[caveman]

Very strange, I'll have a closer look to this file.
No plan for zip support yet, type 0 blocks and a few other requests first.

[caveman]

I know, the latest versions are here on encode.su in the first post.

[Mr_KrzYch00]

Very strange, I'll have a closer look to this file.
No plan for zip support yet, type 0 blocks and a few other requests first.

Actually I know where the problem is. You based huffmix on kzip2gz converter which doesn't include timestamp and extra flags+OS type bytes are zero'ed (that are, in most cases, 2 [compressor used maximum compression - this byte should be either 2 or 4 when CM is 8 - deflate] and 3 [unix OS, 0 means MS-DOS/Win32] respectively). My converter and zopfli KrzYmod puts unix timestamps and exactly 2 and 3 there, since most compressor produce same output. For more information: http://www.onicos.com/staff/iz/formats/gzip.html and https://tools.ietf.org/html/rfc1952 .

It's not a big deal, I can always use kzip2gz, however, huffmix most likely won't work on normal gz files that aren't first converted to ZIP with my zip2gz then back to GZ with Yours kzip2gz, so kinda painful workaround.

PS.: Also You could easly support gzips with filename inside (at least properly FSEEK to deflate streams starting position), if offset 3 (byte 4th) is 8 then read file starting at offset 10 (byte 11th) and read until You encounter NULL byte, after this will be deflate stream.

Part of such code in my gz2zip:

  if(flag[0]==8) {
    gzbuffer = (unsigned char*) malloc (1);
    filenamein = (unsigned char*) malloc (1);
    i=0;
    do {
      if(fread(gzbuffer,1,1,gzfile)) {};
      if(gzbuffer[0]!=0) {
        if(realloc(filenamein,i+1) == NULL) {
          fprintf(stderr,"ERROR: Cannot reallocate memory.\n");
          exit(1);
        }
        memcpy(filenamein+i,gzbuffer,1);
        ++i;
      }
    } while(gzbuffer[0]!=0);

PS2: v13-pre of zopfli KrzYmod has 2 new flags: --cbs (custom block split points) and --cbt (custom block types) so You can easy use it for tests to produce 0 blocks. It's show as example [comparing kzip, zopfli with kzip block splitting and 0 blocks and zopfli + optimizing huffman header] in this post: http://encode.su/threads/2176-Zopfli...ll=1#post43866

[caveman]

Actually I know where the problem is. You based huffmix on kzip2gz converter which doesn't include timestamp and extra flags+OS type bytes are zero'ed (that are, in most cases, 2 [compressor used maximum compression - this byte should be either 2 or 4 when CM is 8 - deflate] and 3 [unix OS, 0 means MS-DOS/Win32] respectively). My converter and zopfli KrzYmod puts unix timestamps and exactly 2 and 3 there, since most compressor produce same output. For more information: http://www.onicos.com/staff/iz/formats/gzip.html and https://tools.ietf.org/html/rfc1952 .

Well not exactly defdb handles this file, huffmix and defdb are sharing the same code for gzip parsing...

[Mr_KrzYch00]

The difference between zip2gz and kzip2gz when the file was converted was 6 bytes beginning at offset 3 and ending at offset 8, being 4 bytes of unix timestamp, extra flags byte and OS type byte. Didn't notice anything else.

[Chortos-2]

I took the liberty to disassemble huffmix to see why it’s choking on gzip files, and I see that when it tries to detect whether the file is gzip as opposed to PNG, it looks for an 8-byte header of exactly 1F8B080000000000, which matches what Mr_KrzYch00 said. Only the first two bytes—or three if you want to report “unknown file type” for hypothetical non-Deflate gzip files—compose the format signature; you shouldn’t expect the rest to be zeros.

[caveman]

I'll fix it.

[Mr_KrzYch00]

cmp     cl, ds:byte_405170[eax]
jnz     short loc_402101
inc     eax
cmp     eax, 8 ; <- change to 4 to reduce iterations to 4 first bytes in GZ files
jl      short loc_4020B0

In hex editor with windows binary (0.6b2 prerelease):
5 314 byte (hex addr: 14C2); change 08 to 04

This reduces number of loop iterations from 8 to 4 making huffmix not check byte 5-8, in turn fixing timestamp being incorrectly compared with NULL bytes in GZ files. As assembly points fread did read 10 first bytes before the check (meaning the offset is now correctly set at start of deflate stream), so there shouldn't be any corruption afterwards.

[SvenBent]

More than 1024 Deflate blocks detected, this is not handled by this version.
More than 1024 Deflate blocks detected, this is not handled by this version.
More than 1024 Deflate blocks detected, this is not handled by this version.

Got these running my optimizition script with defluff delfop/b and huffmix
I believe these are from huffmix choking on my 90mb png files

I still need to 100 verify this

but the fun part is these fiels are optimzig with ect + delfopt
I don't see any issues with my png trials batch script on the same files (Different location of files)

So maybe ect is using more blocks than pngout

I'll look into more but im having to leave my computer for now.
if huffmix could be fixed to take bigger numbers of blocks it would be nice


--
I just verified this is comming form huffmix i was mixing and ect+defl opt optimzied file with the ect+deluff+delfopt optimized version on 95mb

i also verified that using pngout with different combination of defluff or delfopt does not bring up this behavioer. Only the ECT generated ones ( did not test ect without deflopt due to time)