Precomp 0.4

[schnaader]

btw, how large is temporary data written to current directory?

As a rule of thumb, 3 times the decompressed size of the biggest stream in the original file should be enough space. It can get more while in recursion, but usually recursion depth is 1 or 2, so 5-10 times the size should be more than enough in most cases.

The difficult thing here is to estimate the decompressed size of the biggest stream. Archive files like ZIP can be opened and you can look for the largest file to get a good estimation, same for ISO (open with 7-Zip) except you'll have to look for the largest precompressable file and have to guess its decompressed size. PDF files are a special case, they often contain large image streams so the temporary files can even get larger than the original PDF.

Usually, using 512 MB or 1 GB for a RAM-disk should be enough even for precompressing DVD images. I always wanted to test this to see how much the temporary files slow down Precomp, but haven't found a nice RAM-disk solution for Windows so far (although there was I thread about this here which I perhaps should have a look at again).

[Bulat Ziganshin]

the one i use is http://memory.dataram.com/products-a...ftware/ramdisk

[Shelwien]

@echo off
for %%a in (precomp.exe) do set pc=%%~fa
for /r %%a in (*.pcf) do (
  pushd %%~dpa 
  echo [%pc% -r %%~nxa]
  %pc% -r %%~nxa
  popd
)

[schnaader]

the one i use is http://memory.dataram.com/products-a...ftware/ramdisk

Thanks, I just stumbled upon it, too. Works like a charm even with Windows 7. Made some tests with FlashMX.pdf:

Format "(Input files on ..., output files on ..., temporary files on ..., time"
HD = harddisk, RD = ramdisk, ED = external drive (1 TB USB disk)
precompression
  HD HD HD 21 s
  HD RD HD 21 s
  HD HD RD 13 s
  HD RD RD 14 s
  RD RD RD 13 s
  ED ED ED 209 s
  ED ED HD 40 s
  ED HD HD 22 s
recompression
  HD HD HD 3.2 s
  HD RD HD 3.2 s
  HD HD RD 2.9 s
  ED ED ED 6.6 s
  ED ED HD 5.6 s
  ED HD HD 3.4 s

So, indeed, precompression is almost twice as fast when using the RAM-disk for temporary files, recompression uses less temporary files and does only get a little faster.

EDIT: Added times for an external USB drive. Note how extremely slow it is when using the external drive for temporary files, but gets much faster using HD for it and gets close to "normal" speed when using HD for both temporary files and output file.

[Omnikam]

@shelwien Thanks that worked like a charm. So thankyou
@schnaader Thanks for your reply. Script didnt work but i Love Precomp4 havnt had any trouble with it.(works beautifully on sds files)
I tested it on a game i Bought,(for some real life testing)Mafia 2 and with the help of FA compressed to 2.61g. That is amazing my jaw just dropped to the floor!, now if only i could figure out Srep integration i bet it would be smaller still
@schnaader your script does work in fact i "had" to use your solution for one of my projects

[musicdemon]

I recently tried the -precomp argument in FreeARC on a directory with 4GB of pictures with some AVI and THM files scattered in sub-directories. When FreeARC got to the point when it called precomp, however, the command-line portion of the program (being used by precomp) crashed almost immediately. Is this because of unrecognized formats or is it the size? I tried it on a few dozen files (all JPEGs) as a test and it was fine.

[schnaader]

I recently tried the -precomp argument in FreeARC on a directory with 4GB of pictures with some AVI and THM files scattered in sub-directories. When FreeARC got to the point when it called precomp, however, the command-line portion of the program (being used by precomp) crashed almost immediately. Is this because of unrecognized formats or is it the size? I tried it on a few dozen files (all JPEGs) as a test and it was fine.

First of all, I don't know much about the FreeARC part, so the error still could be there (and it could be something simple like a misconfiguration or Precomp not finding packjpg.dll), but I somehow doubt it since some people seem to have used FreeArc/Precomp/SREP... combinations successfully, even on large filesets.

Precomp 0.4 isn't as stable as it could be, main reasons are the new recursion feature and some unknown problems with memory leaks that lead to quite random crashes. Filesize shouldn't be a problem, >2 GB and even >4 GB has been tested, old versions suffered from a slowdown at >4 GB, but there haven't been severe errors.

The best way to find out where the error is would of course be to use only Precomp on all the files which is not that easy because it's a single file archiver at the moment, but not impossible using batch files.

The highest probabily for an error would be large files using recursion, so things like archives containing images/PDFs/other archives.

I don't know if you can use switches for Precomp in FreeARC, but if you can, try -l0 to disable recursion and some variations of -t to only process JPGs or leave out ZIP archives, this will help to find the error. Also, -v enables the debug mode that gives more information about where Precomp crashes in a file.

[musicdemon]

I was able to narrow it down to the files that were causing the issue. Interestingly enough, it wasn't an archive or some rogue corrupted file - it was three AVIs recorded from the same source eons ago (either my old cameraphone or old camera). All have video and audio. Here's the attributes of one of the files:



Any idea as to why the compiler choked on these?

EDIT: Here's the crash dump text

Problem Event Name: APPCRASH
Application Name: precomp04.exe
Application Version: 0.0.0.0
Application Timestamp: 49c51a7f
Fault Module Name: ntdll.dll
Fault Module Version: 6.1.7600.16559
Fault Module Timestamp: 4ba9b29c
Exception Code: c0000005
Exception Offset: 0002e1fe
OS Version: 6.1.7600.2.0.0.256.4
Locale ID: 1033
Additional Information 1: 0a9e
Additional Information 2: 0a9e372d3b4ad19135b953a78882e789
Additional Information 3: 0a9e
Additional Information 4: 0a9e372d3b4ad19135b953a78882e789

[Black_Fox]

Are these videos recorded in MJPEG? That could mean each frame is very similar to JPG (but not the same) and it crashes at PackJPG?

[schnaader]

I'm pretty sure this is a MJPEG issue. You can try using the "precomp -mjpeg-" to disable MJPEG recompression. MJPEG doesn't have DHT headers to save space, so Precomp gets an according error from PackJPG, inserts the standard DHT table and calls PackJPG again. Had no videos so far that led to crashes, but also didn't test that much. Would be nice if you could send one of the files to me as a test file (and to fix the crash).

[musicdemon]

Yes, it turns out they were MJPEGs. Thank goodness for the detailed video analysis by SUPER. I re-encoded them as MPEG-4s in the .mp4 container with Handbrake and precomp was able to digest them. Plus, they were much smaller after converted - guess I need to convert some old .AVI files

EDIT: I PM'd one of the files to you, schnaader.