Precomp 0.4

[schnaader]

Hi!

Finally, Precomp 0.4 is out. Change list:

• New switch -mjpeg for MJPEG recompression support.
• Added recursion (aka multi-pass).
• Added MIME Base64 streams support.
• Added bZip2 streams support.
• Added batch file errorlevels.
• Improved GIF support for partial matches.
• Linked zLib library static - ZLIB1.DLL is not needed anymore.
• Fixed bug that slowed down Precomp for files larger than 4 GB.

Have a look at http://schnaader.info

[Intrinsic]

Awesome

[SvenBent]

Weepeeee

will test soon

Is the multi-pass automatically or... ?

hehe
multipass... leloo dallas mul ti pass.

[John]

Thaaanks!!long time waiting for this..time to test..
PS: linux version soon?

[schnaader]

Is the multi-pass automatically or... ?

Yes, the default setting is recursion depth 10 (should be enough for most files )

For example test with one of the EML files from http://stationeryheaven.com/lund.htm. These are Base64 encoded files containing JPG/GIF files.

Also note the bZip2 stream support. bZip2 does allow some parameter tuning, but this is almost unused, so bZip2 streams require only 1 try and will almost be decompressable completely (so far I haven't found a bz2 file that isn't). Tested with enwik8.bz2 from LTCB:

enwik8.bz2 (29,008,758 bytes) => enwik8.pcf (100,000,028 bytes) => enwik8_.bz2 (identical )

PS: linux version soon?

Indeed, a linux version is being forged. I haven't received a PackJPG library for linux yet so JPG recompression won't be supported, but everything else should work as with the Windows version.

[mstar]

Big thanks!

which mode is best to use with freearc?

[schnaader]

which mode is best to use with freearc?

Depends on the file you want to compress. Usually, just using Precomp without any additional options will do it, but for filetypes not yet supported but using zLib (like SIS, 3DM, zeno - see the filetypes list on my site), slow mode might be better.

By the way, I just reuploaded Precomp because the progress display was a bit jumpy for higher recursion levels (above 2).

Additionally, there have been some changes to the site. Results are now created using Google Charts looking better than the table before, batch error levels are listed and it is possible to download old versions (0.3 - 0.3..

[mstar]

thank you

I have tested it with Level Data for a 3D Game:

\test>precomp.exe Airfield.tar

Precomp v0.4 - ALPHA version - USE FOR TESTING ONLY
Free for non-commercial use - Copyright 2006-2009 by Christian Schneider

Input file: Airfield.tar
Output file: Airfield.pcf

Using PACKJPG.DLL for JPG recompression.

--> packJPG DLL v2.4WIP4 (11/06/2008) by Matthias Stirner <--
More about PackJPG here: http://www.elektronik.htw-aalen.de/packjpg

100.0% - New size: 158227016 instead of 124371456

Done.
Time: 30641 ms

Recompressed streams: 22/22
ZIP streams: 22/22

You can speed up Precomp for THIS FILE with these parameters:
-c8 -m9 -l0

Fast mode does exactly the same for this file, only faster.

(renamed .pcf to .tar.pcf for my test)
compressed the folder, the .tar and the .pcf:

\test>dir

16.03.2009  18:49    <DIR>          Airfield
21.03.2009  18:12       124.371.456 Airfield.tar
21.03.2009  18:13       158.227.016 Airfield.tar.pcf
21.03.2009  18:14        48.460.841 Airfield.arc
21.03.2009  18:16        48.458.280 Airfield.tar.arc
21.03.2009  18:18        34.674.625 Airfield.tar.pcf.arc

(arc option used: -mx)
original size of that freely dowloadable level .rar: 54.19 MB!

awesome improvement!

is there an option to preserve the file extension in the filename( xyz.tar) so that .pcf is added without deleting the old extension (xyz.tar.pcf) ?

[schnaader]

is there an option to preserve the file extension in the filename( xyz.tar) so that .pcf is added without deleting the old extension (xyz.tar.pcf) ?

Not yet (except that you can specify the output filename yourself using the -o option), but I can add it as an additional option to the next version.

[nanoflooder]

Weehee! Great news!
Has the speed of -brute mode improved due to zlib encapsulation?

[schnaader]

Has the speed of -brute mode improved due to zlib incapsulation?

No, zLib speed shouldn't have changed much, so brute mode will still be sloooowww, but I hope I can add support for more stream types like CAB in the next versions so brute mode won't be needed for them anymore.

[nanoflooder]

I'm so looking forward to that Especially CAB (MSZip + LZX), yes. That's gonna seriously improve the compression abilities

[upd] A noob question (got nowhere to test this), does current Precomp support gzip recompression?

[Simon Berger]

Yes it does.

[nanoflooder]

Thank you!

[maadjordan]

Hi!

Finally, Precomp 0.4 is out. Change list:

• New switch -mjpeg for MJPEG recompression support.
• Added recursion (aka multi-pass).
• Added MIME Base64 streams support.
• Added bZip2 streams support.
• Added batch file errorlevels.
• Improved GIF support for partial matches.
• Linked zLib library static - ZLIB1.DLL is not needed anymore.
• Fixed bug that slowed down Precomp for files larger than 4 GB.

Have a look at http://schnaader.info

Great update.. many thanks..

i've tested the "-mjpeg+" option and it reduced compressing time for my test file of (20mb) from 303sec to 65sec - decompression 42sec so great speed ..

but why its not default on?

[Simon Berger]

Seems like there is any sort of bug, because mjpeg is on by default

Did you have the exact same number of found/recompressed streams?

[Black_Fox]

Thanks a lot! Will try it

[LovePimple]

Thanks Christian!

[maadjordan]

Seems like there is any sort of bug, because mjpeg is on by default

Did you have the exact same number of found/recompressed streams?

i have test v0.4 agains same file. first with no option then with "-mjpeg+" added. when added its faster (over 4x faster). thats why i said it needs to be activated always unless it affects existing JPG files which is not true as i tested.

further more it seems that "bZip2" support does not work on this file:

http://heanet.dl.sourceforge.net/sou.../7z465.tar.bz2

please check it out..

[mhajicek]

Thanks for this useful program, Im looking forward to mutliple files/directories support.
This version crashes when trying to process this file (unzip first):
It works only with -l0. (multiple -l0 passes also dont work)

Michal

BTW, any plans to update precomp.dll used by lprepaq/paq8o8pre?

[Bulat Ziganshin]

further more it seems that "bZip2" support does not work on this file:

http://heanet.dl.sourceforge.net/sou.../7z465.tar.bz2

please check it out..

this file is created with 7zip own bzip2 compression code. precomp can only decompress files whose compression algos are built-in into precomp itself, now 7zip bzip2 algo isn't among these

[schnaader]

i have test v0.4 agains same file. first with no option then with "-mjpeg+" added. when added its faster (over 4x faster). thats why i said it needs to be activated always unless it affects existing JPG files which is not true as i tested.

This is strange. I'll investigate this further, but at the moment I can't explain this behaviour. As Simon said, the mjpeg switch is enabled by default and even if it wasn't, it should slow down the process a bit instead of speeding it up.

further more it seems that "bZip2" support does not work on this file:

At least the file can be decompressed (5086720 bytes), although the recompression fails, as you can see when using debug mode. As I mentioned earlier, there are still some ways to modify/tune the bZip2 algorithm that aren't used at the moment because there are 250 (!) possibilities for it. I think I'll do some tests with those bZip2 files and adjust slow mode so it checks those possibilities.

EDIT: This tuning parameter is all about speed, so there's no way to generate other output using the bzip2 library, so it really seems to be some 7-Zip algorithm used here.
See the bzip2 documentation about the workFactor parameter:

Note that the compressed output generated is the same regardless of whether or not the fallback algorithm is used.

This version crashes when trying to process this file (unzip first):
It works only with -l0. (multiple -l0 passes also dont work)

Thanks, I'll have a look at this crash. This could either be some bug with recursion or with gZip streams.

BTW, any plans to update precomp.dll used by lprepaq/paq8o8pre?

Yes, this will happen soon, too. For Prepaq v3, I'll use one of the paq8p releases (although I'm not really sure which one to use, either an optimized paq8p1 or the new paq8p2).

[nanoflooder]

ABBYY uses their own zlib library called AbbyyZlib.dll to compress the Lingvo dictionaries. Unfortunately, it has very different export functions, and the produced file structure seems to be different because Precomp can't decompress it even in -brute mode. Could you please look at it and make a verdict on whether or not is it possible to make something out of this? Thanks in advance
I have attached the library and a sample dictionary (I've chosen a very small one to make it able to be compressed in -brute mode).

[Jed]

Hi there, quick question. I'm a noob when it comes to the inner workings of these things, but is it possible to make the output file self-extracting, or is that just not a viable option through these methods? I like it overall, it's been able to save even more space than I thought, but if it could be self-extracting that would be amazing.

[schnaader]

ABBYY uses their own zlib library called AbbyyZlib.dll to compress the Lingvo dictionaries. Unfortunately, it has very different export functions, and the produced file structure seems to be different because Precomp can't decompress it even in -brute mode. Could you please look at it and make a verdict on whether or not is it possible to make something out of this? Thanks in advance
I have attached the library and a sample dictionary (I've chosen a very small one to make it able to be compressed in -brute mode).

Since their library is not open source and I couldn't find any specifications of the used .lsd format, I doubt that it's possible to decompress those files. I'll try to analyse the format a bit and run some -brute attempts anyway, but I don't expect much from it.

Hi there, quick question. I'm a noob when it comes to the inner workings of these things, but is it possible to make the output file self-extracting, or is that just not a viable option through these methods? I like it overall, it's been able to save even more space than I thought, but if it could be self-extracting that would be amazing.

This is not yet possible, but planned for later versions (expect it to come along with a GUI for Precomp).

[mhajicek]

These PNG files (unzip first) after processing with precomp 0.4 are not restored back correctly from .pcf . Can send you more if needed, but others are bigger (~1MB each).

Michal

[SvenBent]

PreRZM would rock my world

[pat357]

PreRZM would rock my world

It's simple like arc a -mprecomp+rzm ....
Even better : arc a -mprecomp+exe+rep:500m+delta+rzm

Sometimes it's better to discard the exe-filter because RZM seems to have its
own exe-filter.

[SvenBent]

but wouldn't that first make the pcf fiel and then applys RZM compression.
going through the file twice ?

[pat357]

These PNG files (unzip first) after processing with precomp 0.4 are not restored back correctly from .pcf . Can send you more if needed, but others are bigger (~1MB each).

Seems like you found a bug : I tested on one .PNG, v0.37, v0.38 and v0.4 and all produced the same file, but it was indeed NOT identical to the original....