Page 1 of 7 123 ... LastLast
Results 1 to 30 of 191

Thread: Precomp 0.4

  1. #1
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    571
    Thanks
    219
    Thanked 204 Times in 96 Posts

    Precomp 0.4

    Hi!

    Finally, Precomp 0.4 is out. Change list:

    • New switch -mjpeg for MJPEG recompression support.
    • Added recursion (aka multi-pass).
    • Added MIME Base64 streams support.
    • Added bZip2 streams support.
    • Added batch file errorlevels.
    • Improved GIF support for partial matches.
    • Linked zLib library static - ZLIB1.DLL is not needed anymore.
    • Fixed bug that slowed down Precomp for files larger than 4 GB.


    Have a look at http://schnaader.info
    http://schnaader.info
    Damn kids. They're all alike.

  2. #2
    Member
    Join Date
    May 2008
    Location
    England
    Posts
    325
    Thanks
    18
    Thanked 6 Times in 5 Posts
    Awesome


  3. #3
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    878
    Thanks
    50
    Thanked 106 Times in 84 Posts

    Talking

    Weepeeee

    will test soon

    Is the multi-pass automatically or... ?

    hehe
    multipass... leloo dallas mul ti pass.

  4. #4
    Member
    Join Date
    Jul 2008
    Posts
    54
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Thaaanks!!long time waiting for this..time to test..
    PS: linux version soon?

  5. #5
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    571
    Thanks
    219
    Thanked 204 Times in 96 Posts
    Quote Originally Posted by SvenBent View Post
    Is the multi-pass automatically or... ?
    Yes, the default setting is recursion depth 10 (should be enough for most files )

    For example test with one of the EML files from http://stationeryheaven.com/lund.htm. These are Base64 encoded files containing JPG/GIF files.

    Also note the bZip2 stream support. bZip2 does allow some parameter tuning, but this is almost unused, so bZip2 streams require only 1 try and will almost be decompressable completely (so far I haven't found a bz2 file that isn't). Tested with enwik8.bz2 from LTCB:

    enwik8.bz2 (29,008,758 bytes) => enwik8.pcf (100,000,028 bytes) => enwik8_.bz2 (identical )

    Quote Originally Posted by John View Post
    PS: linux version soon?
    Indeed, a linux version is being forged. I haven't received a PackJPG library for linux yet so JPG recompression won't be supported, but everything else should work as with the Windows version.
    Last edited by schnaader; 21st March 2009 at 20:29.
    http://schnaader.info
    Damn kids. They're all alike.

  6. #6
    Member
    Join Date
    Jan 2009
    Location
    Germany
    Posts
    35
    Thanks
    0
    Thanked 0 Times in 0 Posts


    Big thanks!

    which mode is best to use with freearc?

  7. #7
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    571
    Thanks
    219
    Thanked 204 Times in 96 Posts
    Quote Originally Posted by mstar View Post
    which mode is best to use with freearc?
    Depends on the file you want to compress. Usually, just using Precomp without any additional options will do it, but for filetypes not yet supported but using zLib (like SIS, 3DM, zeno - see the filetypes list on my site), slow mode might be better.

    By the way, I just reuploaded Precomp because the progress display was a bit jumpy for higher recursion levels (above 2).

    Additionally, there have been some changes to the site. Results are now created using Google Charts looking better than the table before, batch error levels are listed and it is possible to download old versions (0.3 - 0.3..
    Last edited by schnaader; 21st March 2009 at 21:06.
    http://schnaader.info
    Damn kids. They're all alike.

  8. #8
    Member
    Join Date
    Jan 2009
    Location
    Germany
    Posts
    35
    Thanks
    0
    Thanked 0 Times in 0 Posts
    thank you

    I have tested it with Level Data for a 3D Game:

    Code:
    \test>precomp.exe Airfield.tar
    
    Precomp v0.4 - ALPHA version - USE FOR TESTING ONLY
    Free for non-commercial use - Copyright 2006-2009 by Christian Schneider
    
    Input file: Airfield.tar
    Output file: Airfield.pcf
    
    Using PACKJPG.DLL for JPG recompression.
    
    --> packJPG DLL v2.4WIP4 (11/06/2008) by Matthias Stirner <--
    More about PackJPG here: http://www.elektronik.htw-aalen.de/packjpg
    
    100.0% - New size: 158227016 instead of 124371456
    
    Done.
    Time: 30641 ms
    
    Recompressed streams: 22/22
    ZIP streams: 22/22
    
    You can speed up Precomp for THIS FILE with these parameters:
    -c8 -m9 -l0
    
    Fast mode does exactly the same for this file, only faster.
    (renamed .pcf to .tar.pcf for my test)
    compressed the folder, the .tar and the .pcf:
    Code:
    \test>dir
    
    16.03.2009  18:49    <DIR>          Airfield
    21.03.2009  18:12       124.371.456 Airfield.tar
    21.03.2009  18:13       158.227.016 Airfield.tar.pcf
    21.03.2009  18:14        48.460.841 Airfield.arc
    21.03.2009  18:16        48.458.280 Airfield.tar.arc
    21.03.2009  18:18        34.674.625 Airfield.tar.pcf.arc
    (arc option used: -mx)
    original size of that freely dowloadable level .rar: 54.19 MB!

    awesome improvement!

    is there an option to preserve the file extension in the filename( xyz.tar) so that .pcf is added without deleting the old extension (xyz.tar.pcf) ?
    Last edited by mstar; 21st March 2009 at 21:53.

  9. #9
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    571
    Thanks
    219
    Thanked 204 Times in 96 Posts
    Quote Originally Posted by mstar View Post
    is there an option to preserve the file extension in the filename( xyz.tar) so that .pcf is added without deleting the old extension (xyz.tar.pcf) ?
    Not yet (except that you can specify the output filename yourself using the -o option), but I can add it as an additional option to the next version.
    http://schnaader.info
    Damn kids. They're all alike.

  10. #10
    Member
    Join Date
    Aug 2008
    Location
    Saint Petersburg, Russia
    Posts
    215
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Weehee! Great news!
    Has the speed of -brute mode improved due to zlib encapsulation?
    Last edited by nanoflooder; 21st March 2009 at 23:07.

  11. #11
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    571
    Thanks
    219
    Thanked 204 Times in 96 Posts
    Quote Originally Posted by nanoflooder View Post
    Has the speed of -brute mode improved due to zlib incapsulation?
    No, zLib speed shouldn't have changed much, so brute mode will still be sloooowww, but I hope I can add support for more stream types like CAB in the next versions so brute mode won't be needed for them anymore.
    http://schnaader.info
    Damn kids. They're all alike.

  12. #12
    Member
    Join Date
    Aug 2008
    Location
    Saint Petersburg, Russia
    Posts
    215
    Thanks
    0
    Thanked 0 Times in 0 Posts
    I'm so looking forward to that Especially CAB (MSZip + LZX), yes. That's gonna seriously improve the compression abilities

    [upd] A noob question (got nowhere to test this), does current Precomp support gzip recompression?

  13. #13
    Member
    Join Date
    Oct 2007
    Location
    Germany, Hamburg
    Posts
    408
    Thanks
    0
    Thanked 5 Times in 5 Posts
    Yes it does.

  14. #14
    Member
    Join Date
    Aug 2008
    Location
    Saint Petersburg, Russia
    Posts
    215
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Thank you!

  15. #15
    Member
    Join Date
    May 2008
    Location
    Kuwait
    Posts
    335
    Thanks
    36
    Thanked 36 Times in 21 Posts
    Quote Originally Posted by schnaader View Post
    Hi!

    Finally, Precomp 0.4 is out. Change list:

    • New switch -mjpeg for MJPEG recompression support.
    • Added recursion (aka multi-pass).
    • Added MIME Base64 streams support.
    • Added bZip2 streams support.
    • Added batch file errorlevels.
    • Improved GIF support for partial matches.
    • Linked zLib library static - ZLIB1.DLL is not needed anymore.
    • Fixed bug that slowed down Precomp for files larger than 4 GB.


    Have a look at http://schnaader.info
    Great update.. many thanks..

    i've tested the "-mjpeg+" option and it reduced compressing time for my test file of (20mb) from 303sec to 65sec - decompression 42sec so great speed ..

    but why its not default on?

  16. #16
    Member
    Join Date
    Oct 2007
    Location
    Germany, Hamburg
    Posts
    408
    Thanks
    0
    Thanked 5 Times in 5 Posts
    Seems like there is any sort of bug, because mjpeg is on by default

    Did you have the exact same number of found/recompressed streams?

  17. #17
    Tester
    Black_Fox's Avatar
    Join Date
    May 2008
    Location
    [CZE] Czechia
    Posts
    471
    Thanks
    26
    Thanked 9 Times in 8 Posts
    Thanks a lot! Will try it
    I am... Black_Fox... my discontinued benchmark
    "No one involved in computers would ever say that a certain amount of memory is enough for all time? I keep bumping into that silly quotation attributed to me that says 640K of memory is enough. There's never a citation; the quotation just floats like a rumor, repeated again and again." -- Bill Gates

  18. #18
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts

    Thumbs up

    Thanks Christian!

  19. #19
    Member
    Join Date
    May 2008
    Location
    Kuwait
    Posts
    335
    Thanks
    36
    Thanked 36 Times in 21 Posts
    Quote Originally Posted by Simon Berger View Post
    Seems like there is any sort of bug, because mjpeg is on by default

    Did you have the exact same number of found/recompressed streams?
    i have test v0.4 agains same file. first with no option then with "-mjpeg+" added. when added its faster (over 4x faster). thats why i said it needs to be activated always unless it affects existing JPG files which is not true as i tested.

    further more it seems that "bZip2" support does not work on this file:

    http://heanet.dl.sourceforge.net/sou.../7z465.tar.bz2

    please check it out..
    Last edited by maadjordan; 22nd March 2009 at 13:48.

  20. #20
    Member
    Join Date
    Mar 2009
    Location
    Prague, CZ
    Posts
    60
    Thanks
    27
    Thanked 6 Times in 6 Posts
    Thanks for this useful program, Im looking forward to mutliple files/directories support.
    This version crashes when trying to process this file (unzip first):
    It works only with -l0. (multiple -l0 passes also dont work)

    Michal

    BTW, any plans to update precomp.dll used by lprepaq/paq8o8pre?
    Attached Files Attached Files

  21. #21
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,511
    Thanks
    746
    Thanked 668 Times in 361 Posts
    Quote Originally Posted by maadjordan View Post
    further more it seems that "bZip2" support does not work on this file:

    http://heanet.dl.sourceforge.net/sou.../7z465.tar.bz2

    please check it out..
    this file is created with 7zip own bzip2 compression code. precomp can only decompress files whose compression algos are built-in into precomp itself, now 7zip bzip2 algo isn't among these

  22. #22
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    571
    Thanks
    219
    Thanked 204 Times in 96 Posts
    Quote Originally Posted by maadjordan View Post
    i have test v0.4 agains same file. first with no option then with "-mjpeg+" added. when added its faster (over 4x faster). thats why i said it needs to be activated always unless it affects existing JPG files which is not true as i tested.
    This is strange. I'll investigate this further, but at the moment I can't explain this behaviour. As Simon said, the mjpeg switch is enabled by default and even if it wasn't, it should slow down the process a bit instead of speeding it up.

    Quote Originally Posted by maadjordan View Post
    further more it seems that "bZip2" support does not work on this file:
    At least the file can be decompressed (5086720 bytes), although the recompression fails, as you can see when using debug mode. As I mentioned earlier, there are still some ways to modify/tune the bZip2 algorithm that aren't used at the moment because there are 250 (!) possibilities for it. I think I'll do some tests with those bZip2 files and adjust slow mode so it checks those possibilities.

    EDIT: This tuning parameter is all about speed, so there's no way to generate other output using the bzip2 library, so it really seems to be some 7-Zip algorithm used here.
    See the bzip2 documentation about the workFactor parameter:

    Note that the compressed output generated is the same regardless of whether or not the fallback algorithm is used.

    Quote Originally Posted by mhajicek View Post
    This version crashes when trying to process this file (unzip first):
    It works only with -l0. (multiple -l0 passes also dont work)
    Thanks, I'll have a look at this crash. This could either be some bug with recursion or with gZip streams.

    Quote Originally Posted by mhajicek View Post
    BTW, any plans to update precomp.dll used by lprepaq/paq8o8pre?
    Yes, this will happen soon, too. For Prepaq v3, I'll use one of the paq8p releases (although I'm not really sure which one to use, either an optimized paq8p1 or the new paq8p2).
    Last edited by schnaader; 22nd March 2009 at 21:21.
    http://schnaader.info
    Damn kids. They're all alike.

  23. #23
    Member
    Join Date
    Aug 2008
    Location
    Saint Petersburg, Russia
    Posts
    215
    Thanks
    0
    Thanked 0 Times in 0 Posts
    ABBYY uses their own zlib library called AbbyyZlib.dll to compress the Lingvo dictionaries. Unfortunately, it has very different export functions, and the produced file structure seems to be different because Precomp can't decompress it even in -brute mode. Could you please look at it and make a verdict on whether or not is it possible to make something out of this? Thanks in advance
    I have attached the library and a sample dictionary (I've chosen a very small one to make it able to be compressed in -brute mode).
    Attached Files Attached Files

  24. #24
    Member
    Join Date
    Mar 2009
    Location
    Kentucky, USA
    Posts
    6
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Question

    Hi there, quick question. I'm a noob when it comes to the inner workings of these things, but is it possible to make the output file self-extracting, or is that just not a viable option through these methods? I like it overall, it's been able to save even more space than I thought, but if it could be self-extracting that would be amazing.

  25. #25
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    571
    Thanks
    219
    Thanked 204 Times in 96 Posts
    Quote Originally Posted by nanoflooder View Post
    ABBYY uses their own zlib library called AbbyyZlib.dll to compress the Lingvo dictionaries. Unfortunately, it has very different export functions, and the produced file structure seems to be different because Precomp can't decompress it even in -brute mode. Could you please look at it and make a verdict on whether or not is it possible to make something out of this? Thanks in advance
    I have attached the library and a sample dictionary (I've chosen a very small one to make it able to be compressed in -brute mode).
    Since their library is not open source and I couldn't find any specifications of the used .lsd format, I doubt that it's possible to decompress those files. I'll try to analyse the format a bit and run some -brute attempts anyway, but I don't expect much from it.

    Quote Originally Posted by Jed View Post
    Hi there, quick question. I'm a noob when it comes to the inner workings of these things, but is it possible to make the output file self-extracting, or is that just not a viable option through these methods? I like it overall, it's been able to save even more space than I thought, but if it could be self-extracting that would be amazing.
    This is not yet possible, but planned for later versions (expect it to come along with a GUI for Precomp).
    Last edited by schnaader; 23rd March 2009 at 18:48.
    http://schnaader.info
    Damn kids. They're all alike.

  26. #26
    Member
    Join Date
    Mar 2009
    Location
    Prague, CZ
    Posts
    60
    Thanks
    27
    Thanked 6 Times in 6 Posts
    These PNG files (unzip first) after processing with precomp 0.4 are not restored back correctly from .pcf . Can send you more if needed, but others are bigger (~1MB each).

    Michal
    Attached Files Attached Files
    • File Type: 7z png.7z (586.6 KB, 495 views)
    Last edited by mhajicek; 24th March 2009 at 02:39.

  27. #27
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    878
    Thanks
    50
    Thanked 106 Times in 84 Posts
    PreRZM would rock my world

  28. #28
    Member
    Join Date
    May 2008
    Location
    Antwerp , country:Belgium , W.Europe
    Posts
    487
    Thanks
    1
    Thanked 3 Times in 3 Posts
    Quote Originally Posted by SvenBent View Post
    PreRZM would rock my world
    It's simple like arc a -mprecomp+rzm ....
    Even better : arc a -mprecomp+exe+rep:500m+delta+rzm

    Sometimes it's better to discard the exe-filter because RZM seems to have its
    own exe-filter.

  29. #29
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    878
    Thanks
    50
    Thanked 106 Times in 84 Posts
    but wouldn't that first make the pcf fiel and then applys RZM compression.
    going through the file twice ?

  30. #30
    Member
    Join Date
    May 2008
    Location
    Antwerp , country:Belgium , W.Europe
    Posts
    487
    Thanks
    1
    Thanked 3 Times in 3 Posts

    Arrow

    Quote Originally Posted by mhajicek View Post
    These PNG files (unzip first) after processing with precomp 0.4 are not restored back correctly from .pcf . Can send you more if needed, but others are bigger (~1MB each).
    Seems like you found a bug : I tested on one .PNG, v0.37, v0.38 and v0.4 and all produced the same file, but it was indeed NOT identical to the original....

Page 1 of 7 123 ... LastLast

Similar Threads

  1. Precomp (and Precomp Comfort) in 315 kb
    By Yuri Grille. in forum Data Compression
    Replies: 2
    Last Post: 1st April 2009, 20:40
  2. Precomp 0.3.8
    By schnaader in forum Data Compression
    Replies: 116
    Last Post: 6th March 2009, 10:37
  3. Precomp 0.3.5 is out!
    By squxe in forum Forum Archive
    Replies: 1
    Last Post: 20th August 2007, 15:55
  4. Precomp 0.3.3 is out!
    By squxe in forum Forum Archive
    Replies: 1
    Last Post: 20th July 2007, 18:27

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •