Page 7 of 7 FirstFirst ... 567
Results 181 to 191 of 191

Thread: Precomp 0.4

  1. #181
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    612
    Thanks
    250
    Thanked 240 Times in 119 Posts
    Quote Originally Posted by Bulat Ziganshin View Post
    btw, how large is temporary data written to current directory?
    As a rule of thumb, 3 times the decompressed size of the biggest stream in the original file should be enough space. It can get more while in recursion, but usually recursion depth is 1 or 2, so 5-10 times the size should be more than enough in most cases.

    The difficult thing here is to estimate the decompressed size of the biggest stream. Archive files like ZIP can be opened and you can look for the largest file to get a good estimation, same for ISO (open with 7-Zip) except you'll have to look for the largest precompressable file and have to guess its decompressed size. PDF files are a special case, they often contain large image streams so the temporary files can even get larger than the original PDF.

    Usually, using 512 MB or 1 GB for a RAM-disk should be enough even for precompressing DVD images. I always wanted to test this to see how much the temporary files slow down Precomp, but haven't found a nice RAM-disk solution for Windows so far (although there was I thread about this here which I perhaps should have a look at again).
    http://schnaader.info
    Damn kids. They're all alike.

  2. #182
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,564
    Thanks
    773
    Thanked 687 Times in 372 Posts

  3. #183
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,908
    Thanks
    291
    Thanked 1,271 Times in 718 Posts
    Code:
    @echo off
    for %%a in (precomp.exe) do set pc=%%~fa
    for /r %%a in (*.pcf) do (
      pushd %%~dpa 
      echo [%pc% -r %%~nxa]
      %pc% -r %%~nxa
      popd
    )

  4. #184
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    612
    Thanks
    250
    Thanked 240 Times in 119 Posts
    Quote Originally Posted by Bulat Ziganshin View Post
    Thanks, I just stumbled upon it, too. Works like a charm even with Windows 7. Made some tests with FlashMX.pdf:

    Code:
    Format "(Input files on ..., output files on ..., temporary files on ..., time"
    HD = harddisk, RD = ramdisk, ED = external drive (1 TB USB disk)
    precompression
      HD HD HD 21 s
      HD RD HD 21 s
      HD HD RD 13 s
      HD RD RD 14 s
      RD RD RD 13 s
      ED ED ED 209 s
      ED ED HD 40 s
      ED HD HD 22 s
    recompression
      HD HD HD 3.2 s
      HD RD HD 3.2 s
      HD HD RD 2.9 s
      ED ED ED 6.6 s
      ED ED HD 5.6 s
      ED HD HD 3.4 s
    So, indeed, precompression is almost twice as fast when using the RAM-disk for temporary files, recompression uses less temporary files and does only get a little faster.

    EDIT: Added times for an external USB drive. Note how extremely slow it is when using the external drive for temporary files, but gets much faster using HD for it and gets close to "normal" speed when using HD for both temporary files and output file.
    Last edited by schnaader; 30th September 2010 at 18:27.
    http://schnaader.info
    Damn kids. They're all alike.

  5. #185
    Member
    Join Date
    Sep 2010
    Location
    Australia
    Posts
    46
    Thanks
    0
    Thanked 0 Times in 0 Posts
    @shelwien Thanks that worked like a charm. So thankyou
    @schnaader Thanks for your reply. Script didnt work but i Love Precomp4 havnt had any trouble with it.(works beautifully on sds files)
    I tested it on a game i Bought,(for some real life testing)Mafia 2 and with the help of FA compressed to 2.61g. That is amazing my jaw just dropped to the floor!, now if only i could figure out Srep integration i bet it would be smaller still
    @schnaader your script does work in fact i "had" to use your solution for one of my projects
    Last edited by Omnikam; 4th October 2010 at 05:02.

  6. #186
    Member musicdemon's Avatar
    Join Date
    Oct 2010
    Location
    Massachusetts
    Posts
    5
    Thanks
    0
    Thanked 0 Times in 0 Posts
    I recently tried the -precomp argument in FreeARC on a directory with 4GB of pictures with some AVI and THM files scattered in sub-directories. When FreeARC got to the point when it called precomp, however, the command-line portion of the program (being used by precomp) crashed almost immediately. Is this because of unrecognized formats or is it the size? I tried it on a few dozen files (all JPEGs) as a test and it was fine.

  7. #187
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    612
    Thanks
    250
    Thanked 240 Times in 119 Posts
    Quote Originally Posted by musicdemon View Post
    I recently tried the -precomp argument in FreeARC on a directory with 4GB of pictures with some AVI and THM files scattered in sub-directories. When FreeARC got to the point when it called precomp, however, the command-line portion of the program (being used by precomp) crashed almost immediately. Is this because of unrecognized formats or is it the size? I tried it on a few dozen files (all JPEGs) as a test and it was fine.
    First of all, I don't know much about the FreeARC part, so the error still could be there (and it could be something simple like a misconfiguration or Precomp not finding packjpg.dll), but I somehow doubt it since some people seem to have used FreeArc/Precomp/SREP... combinations successfully, even on large filesets.

    Precomp 0.4 isn't as stable as it could be, main reasons are the new recursion feature and some unknown problems with memory leaks that lead to quite random crashes. Filesize shouldn't be a problem, >2 GB and even >4 GB has been tested, old versions suffered from a slowdown at >4 GB, but there haven't been severe errors.

    The best way to find out where the error is would of course be to use only Precomp on all the files which is not that easy because it's a single file archiver at the moment, but not impossible using batch files.

    The highest probabily for an error would be large files using recursion, so things like archives containing images/PDFs/other archives.

    I don't know if you can use switches for Precomp in FreeARC, but if you can, try -l0 to disable recursion and some variations of -t to only process JPGs or leave out ZIP archives, this will help to find the error. Also, -v enables the debug mode that gives more information about where Precomp crashes in a file.
    http://schnaader.info
    Damn kids. They're all alike.

  8. #188
    Member musicdemon's Avatar
    Join Date
    Oct 2010
    Location
    Massachusetts
    Posts
    5
    Thanks
    0
    Thanked 0 Times in 0 Posts
    I was able to narrow it down to the files that were causing the issue. Interestingly enough, it wasn't an archive or some rogue corrupted file - it was three AVIs recorded from the same source eons ago (either my old cameraphone or old camera). All have video and audio. Here's the attributes of one of the files:

    Click image for larger version. 

Name:	2010-10-04 - 10_44pm.png 
Views:	564 
Size:	35.2 KB 
ID:	1392

    Any idea as to why the compiler choked on these?

    EDIT: Here's the crash dump text

    Problem Event Name: APPCRASH
    Application Name: precomp04.exe
    Application Version: 0.0.0.0
    Application Timestamp: 49c51a7f
    Fault Module Name: ntdll.dll
    Fault Module Version: 6.1.7600.16559
    Fault Module Timestamp: 4ba9b29c
    Exception Code: c0000005
    Exception Offset: 0002e1fe
    OS Version: 6.1.7600.2.0.0.256.4
    Locale ID: 1033
    Additional Information 1: 0a9e
    Additional Information 2: 0a9e372d3b4ad19135b953a78882e789
    Additional Information 3: 0a9e
    Additional Information 4: 0a9e372d3b4ad19135b953a78882e789
    Last edited by musicdemon; 5th October 2010 at 14:36.

  9. #189
    Tester
    Black_Fox's Avatar
    Join Date
    May 2008
    Location
    [CZE] Czechia
    Posts
    471
    Thanks
    26
    Thanked 9 Times in 8 Posts
    Are these videos recorded in MJPEG? That could mean each frame is very similar to JPG (but not the same) and it crashes at PackJPG?
    I am... Black_Fox... my discontinued benchmark
    "No one involved in computers would ever say that a certain amount of memory is enough for all time? I keep bumping into that silly quotation attributed to me that says 640K of memory is enough. There's never a citation; the quotation just floats like a rumor, repeated again and again." -- Bill Gates

  10. #190
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    612
    Thanks
    250
    Thanked 240 Times in 119 Posts
    I'm pretty sure this is a MJPEG issue. You can try using the "precomp -mjpeg-" to disable MJPEG recompression. MJPEG doesn't have DHT headers to save space, so Precomp gets an according error from PackJPG, inserts the standard DHT table and calls PackJPG again. Had no videos so far that led to crashes, but also didn't test that much. Would be nice if you could send one of the files to me as a test file (and to fix the crash).
    http://schnaader.info
    Damn kids. They're all alike.

  11. #191
    Member musicdemon's Avatar
    Join Date
    Oct 2010
    Location
    Massachusetts
    Posts
    5
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Yes, it turns out they were MJPEGs. Thank goodness for the detailed video analysis by SUPER. I re-encoded them as MPEG-4s in the .mp4 container with Handbrake and precomp was able to digest them. Plus, they were much smaller after converted - guess I need to convert some old .AVI files

    EDIT: I PM'd one of the files to you, schnaader.
    Last edited by musicdemon; 5th October 2010 at 15:20.

Page 7 of 7 FirstFirst ... 567

Similar Threads

  1. Precomp (and Precomp Comfort) in 315 kb
    By Yuri Grille. in forum Data Compression
    Replies: 2
    Last Post: 1st April 2009, 19:40
  2. Precomp 0.3.8
    By schnaader in forum Data Compression
    Replies: 116
    Last Post: 6th March 2009, 09:37
  3. Precomp 0.3.5 is out!
    By squxe in forum Forum Archive
    Replies: 1
    Last Post: 20th August 2007, 14:55
  4. Precomp 0.3.3 is out!
    By squxe in forum Forum Archive
    Replies: 1
    Last Post: 20th July 2007, 17:27

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •