Page 1 of 3 123 LastLast
Results 1 to 30 of 64

Thread: gzip-1.2.4-hack - a hacked version of gzip

  1. #1
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,964
    Thanks
    367
    Thanked 341 Times in 134 Posts
    Hi all!

    Let me introduce some fun art by myself!

    gzip-1.2.4-hack is a hacked/improved version of gzip-1.2.4. I played with Deflate and added a slightly improved parsing (lazy matching with two bytes lookahead) which in many cases gives a little bit higher compression than original "-9" mode. Also the gzip-1.2.4 hack keeps input files.

    To use new compression mode, run gzip with the "-9" switch.

    Some testing results:

    world95.txt:
    gzip 1.2.4, -9: 862,955 bytes
    gzip 1.2.4-hack, -9: 857,205 bytes

    fp.log:
    gzip 1.2.4, -9: 1,337,914 bytes
    gzip 1.2.4-hack, -9: 1,319,851 bytes

    bible.txt:
    gzip 1.2.4, -9: 1,176,645 bytes
    gzip 1.2.4-hack, -9: 1,168,229 bytes

    3200.txt:
    gzip 1.2.4, -9: 6,205,632 bytes
    gzip 1.2.4-hack, -9: 6,183,702 bytes

    ENWIK8:
    gzip 1.2.4, -9: 36,445,248 bytes
    gzip 1.2.4-hack, -9: 36,273,716 bytes

    ENWIK9:
    gzip 1.2.4, -9: 322,591,995 bytes
    gzip 1.2.4-hack, -9: 321,050,648 bytes

    Of course, full compatibility was kept. So, why not to get an extra compression?

    Link:
    gzip124hack.zip (61 KB)


  2. #2
    Member
    Join Date
    May 2008
    Location
    Kuwait
    Posts
    312
    Thanks
    27
    Thanked 27 Times in 17 Posts
    did you test 7-zip deflate capabilities as gzip,zip is much tuned for smaller size.. for example

    world95.txt
    7-zip normal 841,725
    7-zip ultra 837,173
    7-zip ultra+word=258 837,081
    7-zip ultra+word=258+pass=20 837,024
    after that i ran

    advdef -z4 836,670

    then

    deflopt 836,667

    so pick the one you like

  3. #3
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,964
    Thanks
    367
    Thanked 341 Times in 134 Posts
    Yep, but 7-Zip's speed... My variant is some sort of compromise. I hope authors of GZIP, finally, add an improved parsing.


  4. #4
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Thanks Ilia!

    Does this version work OK with files larger than 2GB?

  5. #5
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,964
    Thanks
    367
    Thanked 341 Times in 134 Posts
    Quote Originally Posted by LovePimple
    Does this version work OK with files larger than 2GB?
    I dont test it with huge files. If GZIP 1.2.4 supports such files, this version should also. So, I did some research with LZPM and additionally I tested some schemes with Deflate (I hope you saw the results of LZPM 0.09). Unfortunately, Lazy Matching with 2-bytes lookahead is not so efficient as with LZPM, I just expected more compression gain with Deflate. Anyway, the result of this experiments is this version of GZIP.

  6. #6
    Tester

    Join Date
    May 2008
    Location
    St-Petersburg, Russia
    Posts
    182
    Thanks
    3
    Thanked 0 Times in 0 Posts
    //The AdvanceDEFLATE recompression utility
    Where can I download it?

  7. #7
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Quote Originally Posted by encode
    I hope you saw the results of LZPM 0.09
    Yes!

  8. #8
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Quote Originally Posted by encode
    I dont test it with huge files. If GZIP 1.2.4 supports such files, this version should also.
    The reason I asked is because there is a patch for version 1.2.4 to allow handling of large files. Version 1.3.5 (1.3.12) can handle files > 4 GB.

  9. #9
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,964
    Thanks
    367
    Thanked 341 Times in 134 Posts
    If someone ask, I'll compile GZIP 1.3.5 with my parsing.

  10. #10
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Quote Originally Posted by encode
    If someone ask, Ill compile GZIP 1.3.5 with my parsing.
    Awesome!

  11. #11
    Member
    Join Date
    Dec 2006
    Posts
    611
    Thanks
    0
    Thanked 1 Time in 1 Post
    Quote Originally Posted by Squxe
    //The AdvanceDEFLATE recompression utility
    Where can I download it?
    here

  12. #12
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,964
    Thanks
    367
    Thanked 341 Times in 134 Posts
    By the way, DeflOpt rocks! I keep all my data in ZIP files, running this tool recursively saves some disk space!

  13. #13
    Tester

    Join Date
    May 2008
    Location
    St-Petersburg, Russia
    Posts
    182
    Thanks
    3
    Thanked 0 Times in 0 Posts
    thanks BF!

  14. #14
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 778 Times in 485 Posts

  15. #15
    Member
    Join Date
    May 2008
    Location
    Kuwait
    Posts
    312
    Thanks
    27
    Thanked 27 Times in 17 Posts
    the deflopt program is from "black fox" comment

    for the advanceCOM package it includes

    advdef.exe for png,mng,gz
    advpng.exe for png
    advmng.exe for mng
    advzip.exe for zip

    http://advancemame.sourceforge.net/comp-readme.htm l

  16. #16
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,964
    Thanks
    367
    Thanked 341 Times in 134 Posts
    Quote Originally Posted by Matt Mahoney
    Thank you!

  17. #17
    Member Vacon's Avatar
    Join Date
    May 2008
    Location
    Germany
    Posts
    523
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Hello everyone,

    Quote Originally Posted by encode
    If someone ask, Ill compile GZIP 1.3.5 with my parsing.
    Ok, I ask

    Quote Originally Posted by Black_Fox
    here
    Interesting site...!

    Best regards!

  18. #18
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,964
    Thanks
    367
    Thanked 341 Times in 134 Posts
    Looks like a 4 GB patch needed only for gunzip...

    Okay, due to someone's request, I included the modified "deflate.c" source to the gzip-1.2.4-hack package:
    gzip124hack.zip (71 KB)

    So now it hopefully can be compiled on any system, and can be distributed as a GZIP's patch.

  19. #19
    Member Vacon's Avatar
    Join Date
    May 2008
    Location
    Germany
    Posts
    523
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Hello everyone,


    so _someone_ thanks you!

    Edit:
    I got the false impression the 1.3.5 patch was meant.
    Ability to read can be helpful...
    deflate.c was integrated because of GPL, true?

    Best regards!

  20. #20
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Thanks Ilia! Looking forward to v1.3.12 (1.3.5) with your parsing.

  21. #21
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,964
    Thanks
    367
    Thanked 341 Times in 134 Posts
    Quote Originally Posted by Vacon
    deflate.c was integrated because of GPL, true?
    Yep, anyway, the main reason - authors of GZIP can look at this implementation and finally improve parsing in GZIP. Max compression should be really MAX (at least if we talking about efficient implementations without turtle speeds like kzip and 7-Zips LZMA).

    Quote Originally Posted by LovePimple
    Looking forward to v1.3.5 (12) with your parsing.
    Got some problems compiling 1.3.5... Maybe later, or someone recompile the 1.3.5 with my "deflate.c" (this file should be similar to 1.2.4, since the compression of both is the same)

    Just currently I have a little spare time, which I want to spend on more powerful thing - LZPM 0.09!

  22. #22
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Quote Originally Posted by encode
    Got some problems compiling 1.3.5... Maybe later, or someone recompile the 1.3.5 with my "deflate.c" (this file should be similar to 1.2.4, since the compression of both is the same)
    OK!

  23. #23
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,964
    Thanks
    367
    Thanked 341 Times in 134 Posts
    Quote Originally Posted by 4g-patch
    This patch is relative to gzip 1.2.4. It allows uncompressing files
    larger than 4 gigabytes. Files already compressed without the patch
    are correct; the patch is needed only for decompression with gunzip.
    Decompression with zcat outputs the correct data plus an error message
    "length error" that you can ignore.

  24. #24
    Member Vacon's Avatar
    Join Date
    May 2008
    Location
    Germany
    Posts
    523
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Hello everyone,

    Quote Originally Posted by encode
    Just currently I have a little spare time, which I want to spend on more powerful thing - LZPM 0.09!
    me -> <- encode

    Best regards!

  25. #25
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,964
    Thanks
    367
    Thanked 341 Times in 134 Posts

  26. #26
    Member
    Join Date
    May 2008
    Location
    HK
    Posts
    160
    Thanks
    4
    Thanked 25 Times in 15 Posts
    I hope encode will make a hack for zlib too.

  27. #27
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,964
    Thanks
    367
    Thanked 341 Times in 134 Posts
    AFAIK, you can replace ZLIB's deflate.c with modified deflate.c from GZIPHACK and recompile ZLIB, that's all!

  28. #28
    Member
    Join Date
    May 2008
    Location
    HK
    Posts
    160
    Thanks
    4
    Thanked 25 Times in 15 Posts
    Quote Originally Posted by encode
    AFAIK, you can replace ZLIBs deflate.c with modified deflate.c from GZIPHACK and recompile ZLIB, thats all!
    It seems not possible with zlib 1.2.3 as the interfaces are different.

  29. #29
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,964
    Thanks
    367
    Thanked 341 Times in 134 Posts
    Indeed. Anyway, I hope later I'll back to this subject. Note that authors of GZIP/ZLIB was informed about such improvement and most likely, some day will include this feature to their projects.

  30. #30
    Member
    Join Date
    May 2008
    Location
    HK
    Posts
    160
    Thanks
    4
    Thanked 25 Times in 15 Posts
    Quote Originally Posted by encode
    gzip-1.2.4-hack is a hacked/improved version of gzip-1.2.4.
    BTW, besides compressed size, what about compression time?
    hacked version seems a bit faster than the official one downloaded from gzip.org.

Page 1 of 3 123 LastLast

Similar Threads

  1. gzip - Intel IPP
    By M4ST3R in forum Download Area
    Replies: 5
    Last Post: 2nd June 2010, 15:09
  2. Gzip 1.2.4 hack (OpenWatcom compiles)
    By Rugxulo in forum Data Compression
    Replies: 9
    Last Post: 22nd May 2009, 00:17
  3. Parallel implmentation of gzip: pigz
    By nimdamsk in forum Forum Archive
    Replies: 2
    Last Post: 13th March 2007, 20:44

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •