pZLib

[Razor12911]

Pre-ZLIB (pzlib)

This program detects and unpack zlib streams for a certain input (like precomp)
giving you an option to apply a stronger compression such as LZMA.

This only detects zlib streams compressed using default and maximum level
which have headers. (It doesn't detect headerless)

Usage:
pZLib encode/decode [options] inputfile -o outputfile
inputfile/outputfile can be specified as "-" for stdin/stdout

Options:
-t#: Number of threads to use (maximum = 8 )
-t#p: (default = 50) Percentage of threads to use
-v: Verification so that restoration keeps CRC the same
-ai: Prints out additional information

Note:
Input and/or Output can be "-" to signal stdin and/or stdout usage.
Multi threading only works during restoration.
Freearc usage example included

Example:
pzlib encode $$arcdatafile$$.tmp -o $$arcpackedfile$$.tmp

[Dimitri]

you can use stdin and out during compression and decompression !!!

will take 1/2 of the time

[schnaader]

Well done! The concept to process only default zLib streams on maximum compression level first seems a bit odd, but of course it makes very fast encoding possible.

Some general comments:

• Adding a version number to the command line output would be useful
• Allow shortcuts "e"/"d" for "encode"/"decode", too
• Duration output in ms is useful for comparisons, but "human readable" output in hours/minutes/seconds is often preferred
• Open source would be better

pzlib didn't detect any streams in FlashMX.pdf and silesia.zip, so I tested one of my other standard test files, an Ubuntu 15.10 image:

ubuntu-15.10-desktop-amd64.iso

Original: 1,178,386,432 bytes
Precomp 0.4.5
  -cn -intense -d0 (encode): 19 min 30 s, 2,982,177,562 bytes
    -r (decode): 10 min 47 s
  -cn -intense -d0 -zl96,97,98,99 (encode): 14 min 43 s, 2,981,716,646 bytes
pzlib 1100
  encode: 2 min 36 s, 2,974,882,594 bytes
    decode: 10 min 58 s, not identical, output is longer (1,178,403,868 bytes) and differs (first different byte at offset 0x0D05BCDC) 
    decode to stdout: 10 min 45 s
  encode -v: 17 min 29 s, 2,972,136,388 bytes
    decode -t1 stdin, stdout: 16 min 9 s
    decode -t2 stdin, stdout: 12 min 13 s
    decode -t4 stdin, stdout: 9 min 39 s, identical

So encoding without "-v" (verification) is very fast, but decoding fails. Encoding with "-v" is better (decoding is successful), but slower.
Multithreading works, but the baseline is quite slow, so decoding with the max. number of threads is only a bit faster than Precomp.

[Razor12911]

Well, it's just a new program, otherwise, I plan to discard verification and make difflate, which is not going according to plan, it isn't a walk in the park and Multi threading is actually faster than that, but right now limited speed for safety reasons and thank you for the results.

[Razor12911]

Update available


Changes


Threads inversely proportional to speed for stability (new maximum for threads is 4 instead of 8 )
-Speed for multi threading improved
-ms conversion to readable time added as requested
-encode/decode can be shorted to e/d

[Razor12911]

CLS available


Standalone CLS library for Freearc uploaded.
It works without the need of arc.ini or pzlib.exe, it just works by itself, it has compress and decompress code.
You can also paste it in freearc installation directory in bin folder, and use as shown in the pictures.

Drawback is it is quite slow during compression (I'm using ZLIb compiled for delphi and I don't think it is optimized, x64 itself is slower than x86), but decompression is multi threaded, it uses half the number of threads your CPU has.

[Razor12911]

Hi all, can you test this latest build of pZLib.

Program seems to hang sometimes while decoding, just want to see if it hangs from everybody's end.

Note, this is multi threaded and it uses all threads your CPU.

[Razor12911]

Second test, decoding hang fixed.

Note, encoding and decoding use all threads your CPU has.

[anonmous]

Newer version including source code, copied from the other forum page: http://fileforums.com/showthread.php?t=97888