Precomp 0.4

[maadjordan]

ABBYY uses their own zlib library called AbbyyZlib.dll to compress the Lingvo dictionaries. Unfortunately, it has very different export functions, and the produced file structure seems to be different because Precomp can't decompress it even in -brute mode. Could you please look at it and make a verdict on whether or not is it possible to make something out of this? Thanks in advance
I have attached the library and a sample dictionary (I've chosen a very small one to make it able to be compressed in -brute mode).

after i hex-view the file i found this:

deflate 1.2.3 Copyright 1995-2005 Jean-loup Gailly

inflate 1.2.3 Copyright 1995-2005 Mark Adler

I.n.c.\.Z.l.i.b.F.i.l.e...h.......\.S.r.c.\.Z.l.i. b.F.i.l.e...c.p.p

and strange that zlib developers where not mentioned for copyright. May be they but it in the main program..

[schnaader]

These PNG files (unzip first) after processing with precomp 0.4 are not restored back correctly from .pcf . Can send you more if needed, but others are bigger (~1MB each).

Thanks for these files. This seems to be a special bug in processing multi PNGs (PNGs where the zLib data is split into multiple chunks), perhaps related to penalty bytes.
I'll try to fix it - if you have additional images where this happens, I can send you a test version afterwards so you can test if the bug is fixed for all of them.

[mhajicek]

Sure, ready for testing
(for its almost the only thing i can do to help the development because of my poor programming skills)

[schnaader]

I managed to fix both bugs Michal reported.

It seems the crash happened because ZLIB1.DLL isn't used anymore. It looks like I can't compile zLib myself without getting this crashes, even switching compiler optimizations off or compiling ZLIB1.DLL myself didn't help.
So in the next version, ZLIB1.DLL will be needed again until I found out what happens here and how to fix it.

The PNG recompression errors were errors in the multi PNG code. Penalty bytes weren't spread correctly across multiple chunks and sometimes the last chunk length wasn't stored. Those cases occur rarely, so that is why this bug hasn't yet been found in an earlier version.
I send a PM with a developer version to Michal so he can test it on the other PNGs he has. If everything runs fine, I'm looking forward to release Precomp 0.4.1 next week.

[lunaris]

Hello

If possible ,add a plugin interface in the next version and modularise all precompression algos.


(And pleeease , open the source someday)

Thanks

[SvenBent]

will precomp 4.1 have multi file support ( *.* ) and maybe be able to work on same numbers of files as cores in parallel ?

[nanoflooder]

Why would you need the multi-file support in...Precomp? It's so much more useful with batch files

For example: you want to try to decompress all the files and
- delete the .pcf if the file hasn't been decompressed;
- delete the original file if it has;
- do this in current directory and all its subdirectories.

@echo off
@dir /a-d /b /s>"%temp%\precomp.txt"
@cd ..
@pause
@echo Decompressed:
@for /f "usebackq delims=" %%f in ("%temp%\precomp.txt") do (
	rem HidCon.exe might be useful here in case you
	rem don't want to see a lot of blinking windows
	@start /wait cmd /c precomp -slow -o"%%f.pcf" "%%f"
	@if errorlevel 2 (
		@del "%%f.pcf"
	) else (
		@del "%%f"
		@echo   %%f
	)
)
@del "%temp%\precomp.txt"

Then, when you're finished, you would like to compress all the files back, wouldn't you?

@echo off
@dir /a-d /b /s *.pcf>"%temp%\precomp.txt"
@echo Compressed:
@for /f "usebackq delims=" %%f in ("%temp%\precomp.txt") do (
	@start /wait precomp -r -o"%%~dpnf" "%%f"
	@echo   %%f
	@del "%%f"
)
@del "%temp%\precomp.txt"

I personally use [similar to] this code inside some of my current distributions. It really does help to save some weight!

[schnaader]

Hello

If possible ,add a plugin interface in the next version and modularise all precompression algos.

(And pleeease , open the source someday)

Thanks

Opening the source and some kind of interface to add "plug-in" support for own file formats (this is what you meant, isn't it?) will happen when Precomp gets beta, which is not that far away as we're getting close to 0.5 which will be the first beta version.

will precomp 4.1 have multi file support ( *.* ) and maybe be able to work on same numbers of files as cores in parallel ?

I'm pretty sure Precomp 0.4.1 won't support both these features as it will be available soon to fix some 0.4 bugs and crashes, but I want to implement multiple file support next, so it's likely this will be added in 0.4.2. By the way, nanoflooder has a good point here, the latest version returns errorlevels, so it's possible to glue this together yourself. I also think that there are command-line tools to spread commands across different CPU cores, so even that should be possible using batch files.

[SvenBent]

the problem with running the file multiple times by batch is purely performance

is not noticeable on big file. several thousands jpegs the constant loading of precomp does take up a considerable amount of time.


also i wan to be able to do it in several subfoldes as weel but i cannot use sweep for th
sweep'ing af start command make screw up the "" arodun variabel and thne it doesn work with filenames with space
e.g

sweep start for %i in (*.jpg) do precomp "%i"

will not work even thought just
start for %i in (*.jpg) do precomp "%i" works perfectly in each directory


so instead it would be nice to be able to
sweep start precomp *.*

[Skymmer]

Hi!
Finally, Precomp 0.4 is out....

Schnaader, just thank you !!!
I would like to report not the bug but little imprecision. Seems that PreComp stores the name of processed file in low-case. I mean if I use precomp TEST.DAT I got test.dat after recompression. It's not critical but mandatory for some of my tasks. Of course I can use -r -o solution but anyway.

... but I hope I can add support for more stream types like CAB in the next versions ...

It will be just great. As far as I know CAB format used inside of MSI files so make them smaller in distributive will be nice.

ABBYY uses their own zlib library called AbbyyZlib.dll to compress the Lingvo dictionaries. Unfortunately, it has very different export functions, and the produced file structure seems to be different because Precomp can't decompress it even in -brute mode.

You'll not believe me but I thought about it just a few days ago. I suppose DSL Compiler applies not only zlib compression but also some kind of encryption on resulting lsd dics. By the way what Lingvo version you're using? I can't find AbbyyZlib.dll file in my Lingvo 12.

[schnaader]

Schnaader, just thank you !!!
I would like to report not the bug but little imprecision. Seems that PreComp stores the name of processed file in low-case. I mean if I use precomp TEST.DAT I got test.dat after recompression. It's not critical but mandatory for some of my tasks. Of course I can use -r -o solution but anyway.

Conversion to lower case was done to prevent differences in archive when only the filename case differs. This should be indeed corrected, the best behaviour would be to not use the case of the input file parameter but to get the correct case from the file system instead. This will also be important for upcoming Linux versions.

[mhajicek]

To nanoflooder:
Thank you for the posted batch code. Useful, simple, inspired me to learn a bit more about it.
btw tempfile not necessary afaik

[SolVanDyck]

Hi all i just have this problem whenever i precompres a file it always give the same size whatever i use Winrar & 7zip all zip formats i tried nothing seems to work plz can some one post me whats the ideal way to use or the best software and how to precompres it with the best way i would apreciate the help....

[pat357]

Hi all i just have this problem whenever i precompres a file it always give the same size whatever i use Winrar & 7zip all zip formats i tried nothing seems to work plz can some one post me whats the ideal way to use or the best software and how to precompres it with the best way i would apreciate the help....

Try for example a .PDF file and compare the following :
a.pdf -> Winrar -> a.pdf.rar (1)
a.pdf -> Precomp -> a.pdf.pcf -> Winrar -> a.pdf.pcf.rar (2)

compare the size of (1) and (2).
You could do the same using 7zip instead of Winrar.

[SolVanDyck]

Stil the same size nothing has changed

[Fu Siyuan]

Stil the same size nothing has changed

I just tried the program and it is amazing!

Lossless_Compression_Statistical_Model.pdf -- 3.99M
convert it to xxxxxxxxxxxxxxxxxxxxxxxx.pcf -- 9.04M

pdf--> my compressor (csc2) 732k
pcf--> my compressor (csc2) 375k (half reduced)

but the WinRAR works bad on this file. pdf--> rar 709k, pcf---> 698k. I think the small dictionary size (4M) of WinRAR must be the main problem.

[Skymmer]

Stil the same size nothing has changed

First of all can I give advise ? When you ask for help try to be verbose as you can. What type of file you're trying to pre-compress ? What's the feedback from the program ?

In your case, did you try -slow switch ?
Maybe PreComp writes: "None of the given compression and memory levels could be used.
There will be no gain compressing the output file." ?
If so, then PreComp just can't deal with such file.

[SolVanDyck]

I have 3 maps file of a game and i need to recompress them in what the best format do i need to compress them .rar .7z .zip etc...? which ideal u advise me to chose and i need to compress it to maximum? and what the best way to recompress it? and if u could show the command line with the switch that would be prefect and please write the command line in the simplest form so i could copy paste it to a batch file es: precomp.exe filename etc..

[Simon Berger]

I guess you don't know very much about precompression and the data you have.
Precomp extracts zip, gzip, bz2 and some other streams inside a file and reconstructs them on the other way.
It absolutely won't work on every random file and is no magic weapon everywhere. If you want help you have to give us more details.
Use precomp and the -slow switch. Then look what the output says. Dunno whats hard there. If the output says: There will be no gain compressing the output file as Skymmer pointed out it's easy to understand. If it says something like 0/20 streams recompressed there is a small chance that this can be changed in later versions.

[SolVanDyck]

I just want to know how can i use it arent i suppose to compress a file than add it to precomp and when it finishes it size is bigger than the old one then i cmpress it with a more powerful archive than than deflate like freearc and what software to use for deflate people im just confused please help :? :? :?

[Black_Fox]

You don't compress in the beginning, you are supposed to take a file, give it to precomp and compress it after that. (Also you're supposed to search first and ask only if you can't find relevant info yourself, but I guess you'll figure out this one in time )

[nanoflooder]

SolVanDyck, maybe this will help:
Imagine a bunch of files, including dll's and exe's with embedded pictures, jpg's, png's and so on - when you try to compress them with an archiver, it doesn't compress all the precompressed data (such as jpg, png etc.), though the compression ratio of the algorythms used in those files is significantly worse than in most of the today's archivers - it's deflate, gzip, bzip2, instead of lzma, ppmd, proper bwt, you know. So what precomp does is decompresses all that precompressed data, so you can perform superior compression using your own, better archiver, and then afterwards decompress it and recompress all the jpg, gif, png back!

Compression: .jpg (contains metadata + deflated picture) -> [precomp] .jpg.pcf (contains metadata + decompressed picture) -> [7-Zip (for example)] .jpg.pcf.7z (contains well compressed metadata and the original picture)
Decompression: .jpg.pcf.7z -> [7-Zip] .jpg.pcf -> [precomp] .jpg

[Fu Siyuan]

SolVanDyck, maybe this will help:
Imagine a bunch of files, including dll's and exe's with embedded pictures, jpg's, png's and so on - when you try to compress them with an archiver, it doesn't compress all the precompressed data (such as jpg, png etc.), though the compression ratio of the algorythms used in those files is significantly worse than in most of the today's archivers - it's deflate, gzip, bzip2, instead of lzma, ppmd, proper bwt, you know. So what precomp does is decompresses all that precompressed data, so you can perform superior compression using your own, better archiver, and then afterwards decompress it and recompress all the jpg, gif, png back!

Compression: .jpg (contains metadata + deflated picture) -> [precomp] .jpg.pcf (contains metadata + decompressed picture) -> [7-Zip (for example)] .jpg.pcf.7z (contains well compressed metadata and the original picture)
Decompression: .jpg.pcf.7z -> [7-Zip] .jpg.pcf -> [precomp] .jpg

Thank you nanoflooder. it helps me,too. I finally understand what it is used for. It give u a chance to recompress the data with a better algorithm while replacing the old compression method used in the already compressed files.

[nanoflooder]

Yup, exactly.

[SolVanDyck]

thanks nanoflooder allot really but i need the steps only i know it recompresses
but i re compress a zipped file or any file bear with me please..

[nanoflooder]

Let's say you got c:\file.zip to recompress

c:\>precomp -slow -ofile.zip.pcf file.zip

You will get another file, c:\file.zip.pcf, which is a decompressed zip in precomp format. Now you can compress it better

c:\>7z a -mx9 file.zip.pcf.7z file.zip.pcf

The .7z file will weigh much less than the original .zip. So you give the file to your friend along with precomp.exe and packjpg_dll.dll, and to decompress it back you just run

c:\>7z x file.zip.pcf.7z
c:\>precomp -r -ofile.zip file.zip.pcf
c:\>del file.zip.*

[SolVanDyck]

Thanks allot nanoflooder that explains allot man and i even tried it
but it gave a message "non of the given data could be compressed...."
what should i do do i need file larger that 1 GB to work? can i use at the finale compression freearc because i like at allot and if so give the command of freearc
to compress to the maximum.

[nanoflooder]

but it gave a message "non of the given data could be compressed...."

...so why didn't you read that aloud earlier?

what should i do do i need file larger that 1 GB to work?

Nono, Precomp isn't a magic tool - if it doesn't detect anything it can recompress in a file, it won't do that, regardless of the file size.

can i use at the finale compression freearc because i like at allot and if so give the command of freearc to compress to the maximum.

Okay, I allow you to use FreeArc. But only once! Then use gzip forever.

[SolVanDyck]

Gzip what is gzip do u have it download link...

[Vacon]

Hello everyone,

Gzip what is gzip do u have it download link...

I suggest to have a cup of coffee or tea (a BIG one...) and read a site or two...

- http://datacompression.info/
- http://www.maximumcompression.com/programs.php
- http://dir.yahoo.com/Computers_and_I...s/compression/
- http://en.wikipedia.org/wiki/File_compression
- ftp://ftp.elf.stuba.sk/pub/pc/pack/ <= should provide some programs of interest

Just to mention a few...
Be sure to have enough coffee / tea

Best regards!