Results 1 to 9 of 9

Thread: pZLib

  1. #1
    Member Razor12911's Avatar
    Join Date
    May 2016
    Location
    South Africa
    Posts
    32
    Thanks
    58
    Thanked 56 Times in 16 Posts

    pZLib

    Pre-ZLIB (pzlib)

    This program detects and unpack zlib streams for a certain input (like precomp)
    giving you an option to apply a stronger compression such as LZMA.

    This only detects zlib streams compressed using default and maximum level
    which have headers. (It doesn't detect headerless)

    Usage:
    pZLib encode/decode [options] inputfile -o outputfile
    inputfile/outputfile can be specified as "-" for stdin/stdout

    Options:
    -t#: Number of threads to use (maximum = 8 )
    -t#p: (default = 50) Percentage of threads to use
    -v: Verification so that restoration keeps CRC the same
    -ai: Prints out additional information

    Note:
    Input and/or Output can be "-" to signal stdin and/or stdout usage.
    Multi threading only works during restoration.
    Freearc usage example included

    Example:
    pzlib encode $$arcdatafile$$.tmp -o $$arcpackedfile$$.tmp
    Attached Files Attached Files
    Last edited by Razor12911; 25th May 2016 at 00:56.

  2. Thanks (10):

    Bulat Ziganshin (23rd May 2016),comp1 (23rd May 2016),Dimitri (23rd May 2016),gmy (27th May 2017),lorents17 (23rd May 2016),Minimum (22nd May 2016),RamiroCruzo (23rd May 2016),Samantha (23rd May 2016),schnaader (23rd May 2016),Sergey3695 (23rd May 2016)

  3. #2
    Member Dimitri's Avatar
    Join Date
    Nov 2015
    Location
    Greece
    Posts
    48
    Thanks
    21
    Thanked 30 Times in 14 Posts
    you can use stdin and out during compression and decompression !!!

    will take 1/2 of the time

  4. #3
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    615
    Thanks
    255
    Thanked 242 Times in 121 Posts
    Well done! The concept to process only default zLib streams on maximum compression level first seems a bit odd, but of course it makes very fast encoding possible.

    Some general comments:

    • Adding a version number to the command line output would be useful
    • Allow shortcuts "e"/"d" for "encode"/"decode", too
    • Duration output in ms is useful for comparisons, but "human readable" output in hours/minutes/seconds is often preferred
    • Open source would be better


    pzlib didn't detect any streams in FlashMX.pdf and silesia.zip, so I tested one of my other standard test files, an Ubuntu 15.10 image:
    Code:
    ubuntu-15.10-desktop-amd64.iso
    
    Original: 1,178,386,432 bytes
    Precomp 0.4.5
      -cn -intense -d0 (encode): 19 min 30 s, 2,982,177,562 bytes
        -r (decode): 10 min 47 s
      -cn -intense -d0 -zl96,97,98,99 (encode): 14 min 43 s, 2,981,716,646 bytes
    pzlib 1100
      encode: 2 min 36 s, 2,974,882,594 bytes
        decode: 10 min 58 s, not identical, output is longer (1,178,403,868 bytes) and differs (first different byte at offset 0x0D05BCDC) 
        decode to stdout: 10 min 45 s
      encode -v: 17 min 29 s, 2,972,136,388 bytes
        decode -t1 stdin, stdout: 16 min 9 s
        decode -t2 stdin, stdout: 12 min 13 s
        decode -t4 stdin, stdout: 9 min 39 s, identical
    So encoding without "-v" (verification) is very fast, but decoding fails. Encoding with "-v" is better (decoding is successful), but slower.
    Multithreading works, but the baseline is quite slow, so decoding with the max. number of threads is only a bit faster than Precomp.
    http://schnaader.info
    Damn kids. They're all alike.

  5. Thanks (7):

    Bulat Ziganshin (23rd May 2016),gmy (3rd June 2017),lorents17 (23rd May 2016),Minimum (23rd May 2016),RamiroCruzo (23rd May 2016),Razor12911 (23rd May 2016),Samantha (23rd May 2016)

  6. #4
    Member Razor12911's Avatar
    Join Date
    May 2016
    Location
    South Africa
    Posts
    32
    Thanks
    58
    Thanked 56 Times in 16 Posts
    Well, it's just a new program, otherwise, I plan to discard verification and make difflate, which is not going according to plan, it isn't a walk in the park and Multi threading is actually faster than that, but right now limited speed for safety reasons and thank you for the results.

  7. Thanks:

    gmy (3rd June 2017)

  8. #5
    Member Razor12911's Avatar
    Join Date
    May 2016
    Location
    South Africa
    Posts
    32
    Thanks
    58
    Thanked 56 Times in 16 Posts
    Update available


    Changes


    Threads inversely proportional to speed for stability (new maximum for threads is 4 instead of 8 )
    -Speed for multi threading improved
    -ms conversion to readable time added as requested
    -encode/decode can be shorted to e/d

  9. Thanks (5):

    Bulat Ziganshin (24th May 2016),Dimitri (25th May 2016),gmy (3rd June 2017),RamiroCruzo (24th May 2016),schnaader (25th May 2016)

  10. #6
    Member Razor12911's Avatar
    Join Date
    May 2016
    Location
    South Africa
    Posts
    32
    Thanks
    58
    Thanked 56 Times in 16 Posts
    CLS available


    Standalone CLS library for Freearc uploaded.
    It works without the need of arc.ini or pzlib.exe, it just works by itself, it has compress and decompress code.
    You can also paste it in freearc installation directory in bin folder, and use as shown in the pictures.

    Drawback is it is quite slow during compression (I'm using ZLIb compiled for delphi and I don't think it is optimized, x64 itself is slower than x86), but decompression is multi threaded, it uses half the number of threads your CPU has.
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	cls-pzlib.PNG 
Views:	1445 
Size:	92.8 KB 
ID:	4418   Click image for larger version. 

Name:	cls-pzlib2.PNG 
Views:	1263 
Size:	18.9 KB 
ID:	4419  

  11. Thanks (5):

    Bulat Ziganshin (25th May 2016),Dimitri (25th May 2016),gmy (3rd June 2017),Minimum (25th May 2016),RamiroCruzo (25th May 2016)

  12. #7
    Member Razor12911's Avatar
    Join Date
    May 2016
    Location
    South Africa
    Posts
    32
    Thanks
    58
    Thanked 56 Times in 16 Posts
    Hi all, can you test this latest build of pZLib.

    Program seems to hang sometimes while decoding, just want to see if it hangs from everybody's end.

    Note, this is multi threaded and it uses all threads your CPU.
    Attached Files Attached Files

  13. Thanks (3):

    gmy (3rd June 2017),Minimum (5th October 2016),RamiroCruzo (5th October 2016)

  14. #8
    Member Razor12911's Avatar
    Join Date
    May 2016
    Location
    South Africa
    Posts
    32
    Thanks
    58
    Thanked 56 Times in 16 Posts
    Second test, decoding hang fixed.

    Note, encoding and decoding use all threads your CPU has.
    Attached Files Attached Files

  15. Thanks (5):

    gmy (27th May 2017),load (5th October 2016),Minimum (5th October 2016),RamiroCruzo (5th October 2016),~MAK~ (5th October 2016)

  16. #9
    Member
    Join Date
    Mar 2017
    Location
    None
    Posts
    1
    Thanks
    0
    Thanked 3 Times in 1 Post
    Newer version including source code, copied from the other forum page: http://fileforums.com/showthread.php?t=97888
    Attached Files Attached Files

  17. Thanks (3):

    Bulat Ziganshin (15th March 2017),gmy (3rd June 2017),Stephan Busch (15th March 2017)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •