gzip-1.2.4-hack - a hacked version of gzip

[encode]

Hi all!

Let me introduce some fun art by myself!

gzip-1.2.4-hack is a hacked/improved version of gzip-1.2.4. I played with Deflate and added a slightly improved parsing (lazy matching with two bytes lookahead) which in many cases gives a little bit higher compression than original "-9" mode. Also the gzip-1.2.4 hack keeps input files.

To use new compression mode, run gzip with the "-9" switch.

Some testing results:

world95.txt:
gzip 1.2.4, -9: 862,955 bytes
gzip 1.2.4-hack, -9: 857,205 bytes

fp.log:
gzip 1.2.4, -9: 1,337,914 bytes
gzip 1.2.4-hack, -9: 1,319,851 bytes

bible.txt:
gzip 1.2.4, -9: 1,176,645 bytes
gzip 1.2.4-hack, -9: 1,168,229 bytes

3200.txt:
gzip 1.2.4, -9: 6,205,632 bytes
gzip 1.2.4-hack, -9: 6,183,702 bytes

ENWIK8:
gzip 1.2.4, -9: 36,445,248 bytes
gzip 1.2.4-hack, -9: 36,273,716 bytes

ENWIK9:
gzip 1.2.4, -9: 322,591,995 bytes
gzip 1.2.4-hack, -9: 321,050,648 bytes

Of course, full compatibility was kept. So, why not to get an extra compression?

Link:
gzip124hack.zip (61 KB)

[maadjordan]

did you test 7-zip deflate capabilities as gzip,zip is much tuned for smaller size.. for example

world95.txt
7-zip normal 841,725
7-zip ultra 837,173
7-zip ultra+word=258 837,081
7-zip ultra+word=258+pass=20 837,024
after that i ran

advdef -z4 836,670

then

deflopt 836,667

so pick the one you like

[encode]

Yep, but 7-Zip's speed... My variant is some sort of compromise. I hope authors of GZIP, finally, add an improved parsing.

[LovePimple]

Thanks Ilia!

Does this version work OK with files larger than 2GB?

[encode]

Does this version work OK with files larger than 2GB?

I dont test it with huge files. If GZIP 1.2.4 supports such files, this version should also. So, I did some research with LZPM and additionally I tested some schemes with Deflate (I hope you saw the results of LZPM 0.09). Unfortunately, Lazy Matching with 2-bytes lookahead is not so efficient as with LZPM, I just expected more compression gain with Deflate. Anyway, the result of this experiments is this version of GZIP.

[squxe]

//The AdvanceDEFLATE recompression utility
Where can I download it?

[LovePimple]

I hope you saw the results of LZPM 0.09

Yes!

[LovePimple]

I dont test it with huge files. If GZIP 1.2.4 supports such files, this version should also.

The reason I asked is because there is a patch for version 1.2.4 to allow handling of large files. Version 1.3.5 (1.3.12) can handle files > 4 GB.

[encode]

If someone ask, I'll compile GZIP 1.3.5 with my parsing.

[LovePimple]

If someone ask, Ill compile GZIP 1.3.5 with my parsing.

Awesome!

[Black_Fox1]

//The AdvanceDEFLATE recompression utility
Where can I download it?

here

[encode]

By the way, DeflOpt rocks! I keep all my data in ZIP files, running this tool recursively saves some disk space!

[squxe]

thanks BF!

[Matt Mahoney]

I added it to the benchmark. http://cs.fit.edu/~mmahoney/compression/text.html

[maadjordan]

the deflopt program is from "black fox" comment

for the advanceCOM package it includes

advdef.exe for png,mng,gz
advpng.exe for png
advmng.exe for mng
advzip.exe for zip

http://advancemame.sourceforge.net/comp-readme.htm l

[encode]

I added it to the benchmark. http://cs.fit.edu/~mmahoney/compression/text.html

Thank you!

[Vacon]

Hello everyone,

If someone ask, Ill compile GZIP 1.3.5 with my parsing.

Ok, I ask

here

Interesting site...!

Best regards!

[encode]

Looks like a 4 GB patch needed only for gunzip...

Okay, due to someone's request, I included the modified "deflate.c" source to the gzip-1.2.4-hack package:
gzip124hack.zip (71 KB)

So now it hopefully can be compiled on any system, and can be distributed as a GZIP's patch.

[Vacon]

Hello everyone,


so _someone_ thanks you!

Edit:
I got the false impression the 1.3.5 patch was meant.
Ability to read can be helpful...
deflate.c was integrated because of GPL, true?

Best regards!

[LovePimple]

Thanks Ilia! Looking forward to v1.3.12 (1.3.5) with your parsing.

[encode]

deflate.c was integrated because of GPL, true?

Yep, anyway, the main reason - authors of GZIP can look at this implementation and finally improve parsing in GZIP. Max compression should be really MAX (at least if we talking about efficient implementations without turtle speeds like kzip and 7-Zips LZMA).

Looking forward to v1.3.5 (12) with your parsing.

Got some problems compiling 1.3.5... Maybe later, or someone recompile the 1.3.5 with my "deflate.c" (this file should be similar to 1.2.4, since the compression of both is the same)

Just currently I have a little spare time, which I want to spend on more powerful thing - LZPM 0.09!

[LovePimple]

Got some problems compiling 1.3.5... Maybe later, or someone recompile the 1.3.5 with my "deflate.c" (this file should be similar to 1.2.4, since the compression of both is the same)

OK!

[encode]

This patch is relative to gzip 1.2.4. It allows uncompressing files
larger than 4 gigabytes. Files already compressed without the patch
are correct; the patch is needed only for decompression with gunzip.
Decompression with zcat outputs the correct data plus an error message
"length error" that you can ignore.

[Vacon]

Hello everyone,

Just currently I have a little spare time, which I want to spend on more powerful thing - LZPM 0.09!

me -> <- encode

Best regards!

[encode]

[roytam1]

I hope encode will make a hack for zlib too.

[encode]

AFAIK, you can replace ZLIB's deflate.c with modified deflate.c from GZIPHACK and recompile ZLIB, that's all!

[roytam1]

AFAIK, you can replace ZLIBs deflate.c with modified deflate.c from GZIPHACK and recompile ZLIB, thats all!

It seems not possible with zlib 1.2.3 as the interfaces are different.

[encode]

Indeed. Anyway, I hope later I'll back to this subject. Note that authors of GZIP/ZLIB was informed about such improvement and most likely, some day will include this feature to their projects.

[roytam1]

gzip-1.2.4-hack is a hacked/improved version of gzip-1.2.4.

BTW, besides compressed size, what about compression time?
hacked version seems a bit faster than the official one downloaded from gzip.org.