Precomp 0.4

[Skymmer]

Here is another bug. I've tried to PreComp the following Ubuntu image and got the crash. Details below.

PreComp v0.4.0 -o<outfile.pcf> <infile> gives crash at 10.8%
Running it with -v gives the last line:

Possible zLib-Stream in GZip found at position 79008154, windowbits = 15
Can be decompressed to 5766 bytes
Identical recompressed bytes: 2489
Identical decompressed bytes: 5766 of 5766
Real identical bytes: 2489
Best match with compression level 9: 2489 bytes, decompressed to 5766 bytes
Recursion start - new recursion level 1
No recursion streams found
Recursion end - back to recursion level 0

PreComp v0.4.0 -l0 -o<outfile.pcf> <infile> also crash at about 10.7% but interesting thing:
PreComp v0.4.0 -v -l0 -o<outfile.pcf> <infile> somehow allowes to go beyond this error (300 MB output instead of 80 MB) but anyway crashes:

Possible zLib-Stream in GZip found at position 303384380, windowbits = 15
Can be decompressed to 63125 bytes
Identical recompressed bytes: 5200
Identical decompressed bytes: 14659 of 63125
Real identical bytes: 5199
Identical recompressed bytes: 5200
Identical decompressed bytes: 14659 of 63125
Real identical bytes: 5199
Identical recompressed bytes: 5200
Identical decompressed bytes: 14659 of 63125
Real identical bytes: 5199
Best match with compression level 9: 5199 bytes, decompressed to 14659 bytes
Penalty bytes were used: 5 bytes

v0.3.8 works fine by the way.

[schnaader]

Here is another bug. I've tried to PreComp the following Ubuntu image and got the crash. Details below.

[...]

There is some problem with using the zLib code instead of the DLL (which is one of the differences between 0.3.8 and 0.4, I think we can outrule recursion because you also tested -l0) that leads to a crash - not often, but reproducible and with weird variations (your -l0 -v test).

I tried compiling and using the zLib code with various optimizing switches and even tried compiling DLLs myself - nothing but using the original ZLIB1.DLL could avoid the crashes.

This is why the next version will contain ZLIB1.DLL again. Perhaps the source of this problem will be found later, but at the moment this is the best solution to get to a more stable version.

[SolVanDyck]

I tried to precom a file but when evrytime it reaches 17.2 it gives me
"error 00000017.dat cannot read" what could be the probleme....

[schnaader]

I tried to precom a file but when evrytime it reaches 17.2 it gives me
"error 00000017.dat cannot read" what could be the probleme....

Hmm.. what's the exact error message? This could be some problem with the temporary files, but could also be anything else, there's no error message like the one you're describing in Precomp. So either it's a PackJPG error message (which should be prefixed by "PackJPG error") and you can fix it using the switch -t-j or it's some other error with a certain stream which can perhaps be fixed by using the debug switch -v and then ignoring the position of the last stream.

[SolVanDyck]

when i used a precomp with .zip files it precompreses it but not effective comparing with .rar so i tried both formats the one with .rar gave me this error although i used rar in all of my previous precomps worked perfectly only on this file and even better that .zip format.

[Skymmer]

PreComp 0.3.8 crashes with this tar file. Crash appears in both default and slow modes. Running with -v gives last line:

Possible GIF found at position 1769472

Interesting thing that 0.4.0 works but only with -v switch. I suppose its once again something related to ZLIB problem but why then 0.3.8 crashes ?

[Jan Ondrus]

PreComp 0.3.8 crashes with this tar file. Crash appears in both default and slow modes. Running with -v gives last line:

Possible GIF found at position 1769472

Interesting thing that 0.4.0 works but only with -v switch. I suppose its once again something related to ZLIB problem but why then 0.3.8 crashes ?

Looks like "sorry.gif" file is causing crash for some reason.

[Skymmer]

Yes, more exactly speaking Aiwan\light_skin\sorry.gif but interesting thing - this file itself (not TAR-ed) and your TAR are not crashing on my system.

[Jan Ondrus]

Yes, more exactly speaking Aiwan\light_skin\sorry.gif but interesting thing - this file itself (not TAR-ed) and your TAR are not crashing on my system.

Yes.
And same problem is with Aiwan\standart\sorry.gif
If removed this 2 files from kolobok_full_archive.tar archive it won't crash.

[SolVanDyck]

I have added -v switch and it worked not giving me crashes anymore
just out of curiosity when will the next version appear

[schnaader]

I have added -v switch and it worked not giving me crashes anymore

Yes, removing JPEG recompression and enabling verbose mode seem to be the best things to avoid crashes with 0.4.0, although I've got no idea why enabling verbose mode helps - there must be some memory leak or similar strange bug.

just out of curiosity when will the next version appear

When it's done - no, seriously, I think it won't take too long from now - to sweeten the time until the next release, here's some output from the current development version:

Precomp v0.4.1 - DEVELOPMENT version - NO OFFICIAL RELEASE!
Free for non-commercial use - Copyright 2006-2009 by Christian Schneider

Switches (and their <default values>):
  [...]
  zl[1..9][1..9] zLib levels to try for compression (comma seperated) <all>
  c[-b]        Compression method to use, - = none, b = bZip2 <b>

---

Precomp v0.4.1 - DEVELOPMENT version - NO OFFICIAL RELEASE!
Free for non-commercial use - Copyright 2006-2009 by Christian Schneider

Input file: FlashMX.pdf
Output file: FlashMX.pcf

[...]

100.0% - New size: 3013398 instead of 4526946

Done.
Time: 1 minutes, 18 seconds

Recompressed streams: 201/223
PDF streams: 197/219
JPG streams: 4/4

You can speed up Precomp for THIS FILE with these parameters:
-zl65 -l0

---

Precomp v0.4.1 - DEVELOPMENT version - NO OFFICIAL RELEASE!
Free for non-commercial use - Copyright 2006-2009 by Christian Schneider

Input file: FlashMX.pcf
Output file: FlashMX.pdf

[...]

Time: 8 seconds, 502 milliseconds

Notable changes:

• Time output is more human-readable
• Switches -c and -m have been merged to the "zLib levels" switch -zl speeding up the compression a bit more because it's more exact
• bZip2 "compression-on-the-fly" which is used by default now

Don't take times too serious - crappy 800 MHz AMD used.
There are some bugs left - recursion in Base64 streams isn't handled correctly if compression-on-the-fly is enabled, but I'm confident to fix this and release the new version soon.

[Bulat Ziganshin]

precomp+freearc+innosetup is pretty popular these days for huge program installations. unfortunately, there is big problem for their usability - precomp runs as external process. can you make one of the following:

1. support in precomp decompression stdin-to-stdout mode
2. make available some precomp.dll or .lib
3. make precomp decompression open-source

?

[schnaader]

precomp+freearc+innosetup is pretty popular these days for huge program installations. unfortunately, there is big problem for their usability - precomp runs as external process. can you make one of the following:

1. support in precomp decompression stdin-to-stdout mode
2. make available some precomp.dll or .lib
3. make precomp decompression open-source

?

Hmm... a precomp.dll/lib is ready and you could use it. Paq8o8pre and lprepaq work that way.

Feel free to download paq8o8pre/lprepaq and use precomp.dll from it - you'll see how to use it in the source files. For a lib file, there's libprecomp.a included although I don't know this works for non-gcc compilers, too. There's a precomp.h, too, that provides a Switches class to init/change Precomp parameters. The methods provided are:

bool precompress_file(char* in_file, char* out_file, char* msg, Switches switches);
bool recompress_file(char* in_file, char* out_file, char* msg, Switches switches);

The only thing I'd need to change in that case would perhaps be the output - at the moment, it outputs the progress to stdin every now and then, I think it won't be possible or at least hard to bypass this to a GUI or progress bar display. So I could implement an optional callback or something else.

Making precomp decompression open-source is something I'm also thinking of, but I fear it's not going to happen soon, so using DLL would be a better solution at the moment.

[Bulat Ziganshin]

using such API doesn't make much meaning. in both cases you have external dll/exe and need to write entire input to the file, wait until precomp finishes and read it back. what i mean is the way to run precomp "online". simplest method for you may be just stdin-to-stdout mode, although other ways of course would be better for users

Making precomp decompression open-source is something I'm also thinking of, but I fear it's not going to happen soon,

why? i thought what your IP is compression code, although i may be easily wrong

[schnaader]

using such API doesn't make much meaning. in both cases you have external dll/exe and need to write entire input to the file, wait until precomp finishes and read it back. what i mean is the way to run precomp "online". simplest method for you may be just stdin-to-stdout mode, although other ways of course would be better for users

Hmm.. actually, stdin-to-stdout could be quite easy to implement, I could try to implement it in the upcoming version. So if I understand you correctly, you'd want to give a switch, say "-stdout", which makes Precomp use stdin and stdout instead of files, right?

why? i thought what your IP is compression code, although i may be easily wrong

Yes, actually recompression is not a big secret, so it can be open-sourced, it's just that it would be additional source code to write and maintain and it doesn't have a high priority for me at the moment. OTOH, I might better open source it now - doing so could get more complex with future versions, and there's always a chance of people improving the recompression code if it's open...

[Bulat Ziganshin]

So if I understand you correctly, you'd want to give a switch, say "-stdout", which makes Precomp use stdin and stdout instead of files, right?

yes

Yes, actually recompression is not a big secret, so it can be open-sourced, it's just that it would be additional source code to write and maintain and it doesn't have a high priority for me at the moment.

making it OSS may (or may not) help to spread precomp. i.e. some programs may start to use it since decompression may be incorporated inside these programs. for example, if precomp decompression will be inside freearc, it will become possible to extract any .arc archives on any platforms w/o external dll/executables, so precomp-enabled .arc would be no more seocnd-class citizens. of cpurse, it may be not what you want

about technical issues - it's not necessary to make separate sources version. just move compression code that includes your IP into separate files and don't include these files into source distribution. i think it's no much problem if your source distribution will include minor code that don't required for extraction

[SvenBent]

Precomp04 corrupts data when working on network drives (pcf->org).

The same files copied to hard drive first has perfect md5 checksum after precomp.

Rep has shown similar behavior but no other programs then those two.
I've tested for data corruption on network but found nothing...


-- edit --
It is .jpg files I've seen this happen on. dunno if its also on other files
I will test with packjpg tonight.

[schnaader]

Precomp04 corrupts data when working on network drives (pcf->org).

The same files copied to hard drive first has perfect md5 checksum after precomp.

Rep has shown similar behavior but no other programs then those two.
I've tested for data corruption on network but found nothing...


-- edit --
It is .jpg files I've seen this happen on. dunno if its also on other files
I will test with packjpg tonight.

Could you explain "network drives" a bit? I assume it's a local network drive under Windows. Strange that it's the recompression that uses less temporary files and less drive access. Have you tried the other way (org->pcf), too? Also, were source and destination both network points or was the destination the hard drive?

PackJPG would be an explanation, in that case using -t-j and retesting recompression with this file would succeed.

Strange one...

[SvenBent]

Could you explain "network drives" a bit?

YES it is network path mounted as a drive.

i contain all my data files an a server where the path is mounter as drive X on my main machine.

Here is my approach

I have a couple of hundred jpegs files.
i use quicksfv to make a md5 chekcsum file
form CLI i run this: for %i in (*.jpg) do precomp %i
I erase alle the jpeg files
I store alle the pcf files in a 7-contain
i add the md5 chekcsum file, precomp, packjpg.dll adn extract.bat into the 7-zip file.
i extract the file to a new subfolder and run extract.bat which makes all the pcf back to jpg.
i run md5 checksum (OK)
i put the 7-zip file into ma network drive
Erase all my previous folder from my main machine.


Her is the problem at hand

i decompres the 7-zip into a sub folder in the X: drive
i run extract.bat from withing this subfolder.
some of the precomp instance crashes
i run md5 cheksum verification (some bad files)

i delete the subfolder again
transfer the 7-zip file back to main machine
decompress into subfolder at my main machine.
run extract.bat
no crashes
md5 verification OK


i try once more on the network drive. and it still fails however with other files



Here is my extrac bat
@ECHO OFF

ECHO for %%%%f in (%%1.pcf) do Precomp.exe -r "%%%%f" > DCMPTPM.bat
ECHO del "%%1.pcf" >> DCMPTPM.bat
ECHO exit >> DCMPTPM.bat
start DCMPTPM.bat 0??
start DCMPTPM.bat 1??
start DCMPTPM.bat 2??

:checkPCF
ping 127.255.255.255 -n 1 -w 255 > nul
if exist *.pcf goto checkPCF

del DCMPTPM.bat
del precomp.exe
del packjpg_dll.dll
del %0



as you can se i run multiple precomps in parallel. but it works perfect on local drive


i've seen this on is several different collection of jpegs. i believe the inputs file is not the issue but maybe there is some timings issues.

[schnaader]

Hmm.. I'd check the "parallel" thing first, although this shouldn't be a difference for local/network, but who knows. For example, output all the pcf filenames to a debug file, perhaps you try to recompress the same files twice at the same time or something like this.

If even a serial approach doesn't work, you should be able to find out which files work and which crash, but I fear that could be different each time.

Additionally, you could check the md5 sums of the PCF files before extraction, perhaps they aren't transferred correctly/completely.

[SolVanDyck]

could u make a solution for temp files because they are kind of anoying when u refresh ur recompression folder i gives alot of files..can't u just make a one tmp file or something.
and one more thing when a failed recompression my system freezes up and i have like to restart my explorar but after a successful recompression it doesn't freez.

[SvenBent]

Additionally, you could check the md5 sums of the PCF files before extraction, perhaps they aren't transferred correctly/completely.

i dont keep the md5 checksum of the pcf.files as 7-zip has crc32 values of these when i decompress the 7-zip there are no crc32 errors.

all the files are named byt 3 numbers i cant se any possibility where 0??.jpg 1??.jpg and 2??.jpg would ever collide.


I will however test some more as soon as i have time. i dont know if it will be before this weekend.

[Skymmer]

I've found a strange bug in PreComp regarding the JPEG processing. The story is following. I was doing the JPEG compression test with one testbed which consists of 295 files. All these files have 2816x2112 or 2112x2816 resolution depending on orientation. PreComp v0.4.0 perfectly works on files with 2816x2112 resolution but doesn't work for 2112x2816 res. files. No no, output files are created but PreComp doesn't use JPEG compression for them. Here is the typical message for such files:

Possible JPG found at position 0, length 21274
packJPG error: unexpected end of data encountered
No matches
Possible JPG found at position 4424, length 15522
Best match: 15522 bytes, recompressed to 13951 bytes
Possible zLib-Stream (slow mode) found at position 216873, windowbits = 8
Can be decompressed to 22 bytes
Less than 32 bytes, skipping.
Possible zLib-Stream (slow mode) found at position 781368, windowbits = 9
Can be decompressed to 13 bytes
Less than 32 bytes, skipping.
New size: 1312361 instead of 1313885

More interesting is that if I pass such files through some Metadata cleaner like PureJPEG then everything becomes fine. I'll better illustrate it with this:

DSC01499.jpg           1 370 536
DSC01499.pcf           1 067 717

DSC01500.jpg           1 313 885
DSC01500.pcf           1 312 361

DSC01500_NoMETA.jpg    1 292 613
DSC01500_NoMETA.pcf    1 026 400

I uploaded these JPEGs just in case.
http://skymmer.narod.ru/misc/precomp_jpeg_report.rar

[schnaader]

I've found a strange bug in PreComp regarding the JPEG processing [...]

I'll have a look into it. My first quick guess is that this is a case where Precomp's JPEG detection fails. That doesn't occur that often anymore, but it still happens and this could be one of those cases.

[SvenBent]

i did some more test re gargisn my previeus issues when runnig multiple precomps at once


Lokal .7z -> lokal folder -> multi precompt = crashes with packjpeg.dll
LANl .7z -> lokal folder -> multi precompt = crashes with packjpeg.dll
Lan .7z -> lokal 7.z -> lokal foldere -> milti precomp = Crashes with Packjpg.dll
Lokal .7z -> LAn folder -> multi precompt = crashes with packjpeg.dll


When running single precomp everything decodes perfect and md5 chekc matches

I'm running 11 precompt in parralels for this test

Sadly i cannot send you these test files as it contains private pictures

But i'll se if i can make it with non private pictures


THis is my ExtractPCF.bat

@ECHO OFF



ECHO for %%%%f in (%%1.pcf) do Precomp.exe -r "%%%%f" > DCMPTPM.bat
ECHO del "%%1.pcf" >> DCMPTPM.bat
ECHO exit >> DCMPTPM.bat
start DCMPTPM.bat 00?
start DCMPTPM.bat 01?
start DCMPTPM.bat 02?
start DCMPTPM.bat 03?
start DCMPTPM.bat 04?
start DCMPTPM.bat 05?
start DCMPTPM.bat 06?
start DCMPTPM.bat 07?
start DCMPTPM.bat 08?
start DCMPTPM.bat 09?
start DCMPTPM.bat 10?



:checkPCF
ping 127.255.255.255 -n 1 -w 255 > nul
if exist *.pcf goto checkPCF

del DCMPTPM.bat
del precomp.exe
del packjpg_dll.dll
del %0

[Mushoz]

Thanks for this great little program! I have a question. I am currently running precomp 0.4 on a >4gb file on the desktop, and it's causing explorer.exe to constantly use 25% of my CPU (quad core, so it's completely taking up 1 core). I've confirmed it is precomp causing the CPU usage of explorer to get so high. I'm not sure if it has anything to do with me testing the program on the desktop at the moment, so I'll check that out once it's done with this file.

Thanks again for this program. I'm really looking forward to a new version

[Mushoz]

I've ran a test by using precomp in a folder on the root of my HDD and I'm still getting the high CPU usage on explorer.exe. I assume this has to do with the fact it's doing a lot of file access, because of the fact it's mostly using temporary files instead of the RAM. I assume the explorer usage will go down with the new version, once it uses the RAM instead of temp files? Thanks

[Mushoz]

Is there any chance of this program becoming multithreaded in the future? That would be awesome Thanks!

[Bulat Ziganshin]

bug(?) report with 0.4: http://freearc.org/download/testing/af_caves.ff processed with -slow suugest to use fast mode and concrete -l and so on switches. but actually w/o -slow this file cannot be precomped, with ot without suggested switches. but may be i misunderstand precomp suggestions?

[schnaader]

Is there any chance of this program becoming multithreaded in the future? That would be awesome Thanks!

Not in the near future, but yes, it will become multithreaded at some point. It's predestined for that as there usually are many streams that can be processed independently, so it should get almost linear speedup.

bug(?) report with 0.4: http://freearc.org/download/testing/af_caves.ff processed with -slow suugest to use fast mode and concrete -l and so on switches. but actually w/o -slow this file cannot be precomped, with ot without suggested switches. but may be i misunderstand precomp suggestions?

I can't download the file, but I think I know what the problem is. Actually, slow and fast are bad names for the switches as they make you think it's either fast or slow, but actually they can be combined. A better naming would be "fast" mode and "intense" mode or something like this.

So the parameters you need should be "-slow -fast -l [...]"