Results 1 to 23 of 23

Thread: Precomp v0.3.8 >2GB test version

  1. #1
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    551
    Thanks
    206
    Thanked 182 Times in 87 Posts

    Precomp v0.3.8 >2GB test version

    I just managed to compile a test version of Precomp that could be able to process files > 2 GB, it would be nice if someone could check this, especially for files even greater than 4 GB.

    Get the test version here: http://schnaader.info/precomp038_2gb_test_v3.zip
    (Edit: Changed link to new v2 version, also see thread below)
    (Edit2: v3 is out )

    Use only for testing, I'm pretty sure there'll be some bugs in this test version!
    Last edited by schnaader; 6th June 2008 at 20:15.
    http://schnaader.info
    Damn kids. They're all alike.

  2. #2
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    870
    Thanks
    47
    Thanked 105 Times in 83 Posts

    Talking

    Since i was the one to requeste/mention it public. i will gladly do some >2gb and >4gb file test.

    going to start in about now...

  3. #3
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    870
    Thanks
    47
    Thanked 105 Times in 83 Posts
    Since you are still using ~tempX.dat files i guess that i still can't use it on several files in the same directory ?.

    cant we make the files kinda random. or use the same name as the intput file ?

    it would really sped up my compression/decompression as i would be able to utilize more cores.

  4. #4
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    870
    Thanks
    47
    Thanked 105 Times in 83 Posts
    i just used precomp on a 408gb .zip file.
    the file was made with 7-zip fast and had the contents o three copies of my warcraft3 folder and somde pdf documents.
    the compression raita was around 95%

    the originale file and the precomped-restored file had tge same md5 checksum.


    but here is my concerne., alle the three files was all 4.08gb in size

    test.pcf - 4387695058
    test.zip.restored - 4387673979
    test.zip.org.original - 4387673979

    the PCF file only grew with 21079 bytes... i had thougth that the .pcf file would grow back to something like size of the uncompressed data.


    I'm going to see what kind of better compression the pcf would give me compared to the originale zip

  5. #5
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    551
    Thanks
    206
    Thanked 182 Times in 87 Posts
    Quote Originally Posted by SvenBent View Post
    Since you are still using ~tempX.dat files i guess that i still can't use it on several files in the same directory ?.

    cant we make the files kinda random. or use the same name as the intput file ?

    it would really sped up my compression/decompression as i would be able to utilize more cores.
    OK, I think "same name as the input file" sounds good, I'll add this.

    Quote Originally Posted by SvenBent
    i just used precomp on a 408gb .zip file.
    the file was made with 7-zip fast and had the contents o three copies of my warcraft3 folder and somde pdf documents.
    the compression raita was around 95%

    the originale file and the precomped-restored file had tge same md5 checksum.

    ...... snip ......

    the PCF file only grew with 21079 bytes... i had thougth that the .pcf file would grow back to something like size of the uncompressed data.


    I'm going to see what kind of better compression the pcf would give me compared to the originale zip
    I guess only some of the .ZIP contents could be recompressed, since Precomp performs bad on big ZIP streams (> 500 KB or so). Compression will not be much better, I guess. The next version will perhaps feature multiple file support, so then you can just precompress the original data without creating a ZIP before.
    However, the MD5 checksum is the same, that sounds great. Thanks a lot for testing!
    http://schnaader.info
    Damn kids. They're all alike.

  6. #6
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    870
    Thanks
    47
    Thanked 105 Times in 83 Posts
    just to beat you and advance.

    when doing multiple file support, could you make it SMP aware? so that it would work on the same numbers of files as cores in parallel.
    That would be so cool

    and to correct myself,
    it was a 4.08GB file. not 408GB


    going for some 6gb and 8 gb iso files next.

  7. #7
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    870
    Thanks
    47
    Thanked 105 Times in 83 Posts
    just made a 3.76GB zip file with 7-zip (fast)

    it sucefully got precomped to a PCF file.

    but when restorin the originale file made this error

    Code:
    C:\>precomp.exe -r "Librarian II.pcf"
    
    Precomp v0.3.8 - > 2GB DEVELOPMENT version - NO OFFICIAL RELEASE!
    Free for non-commercial use - Copyright 2006,2007 by Christian Schneider
    
    Input file: Librarian II.pcf
    Output file: librarian ii.zip
    
    Using PACKJPG.DLL for JPG recompression.
    
    --> packJPG DLL v2.3b (20/12/2007) by Matthias Stirner <--
    More about PackJPG here: http://www.elektronik.htw-aalen.de/packjpg
    
     16.7%
    ERROR: There is not enough space on disk
    I tested the zip file with 7-zip test. and it was ok.
    then i made a new pcf file.
    but trying to restore the original file back the same error occurred.

    i still have the zip and pcf file if you want them.



    ups and BTW i had 50GB free space one the drive

  8. #8
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    551
    Thanks
    206
    Thanked 182 Times in 87 Posts
    Quote Originally Posted by SvenBent View Post
    Code:
     16.7%
    ERROR: There is not enough space on disk
    I tested the zip file with 7-zip test. and it was ok.
    then i made a new pcf file.
    but trying to restore the original file back the same error occurred.

    i still have the zip and pcf file if you want them.

    ups and BTW i had 50GB free space one the drive
    The best would be if you'd try to restore it again and use the -v debug switch. This would reveal which stream type led to the crash. I'll have a look at the code today and run some tests on various stream types, so I'll probably find the bug.
    http://schnaader.info
    Damn kids. They're all alike.

  9. #9
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    551
    Thanks
    206
    Thanked 182 Times in 87 Posts
    There was a bug when processing penalty bytes that led to file positions that were very big, so that's why "Not enough space on disk" was shown.

    I fixed the bug and uploaded a new test version. "v2" was added both to the archive name and the program message to be able to keep both versions apart, the old test version isn't available anymore.

    New test version (v2): http://schnaader.info/precomp038_2gb_test_v2.zip

    (Edit: Just reuploaded a version that creates unique temporary files, so running multiple Precomp tasks is now possible even in the same directory)
    Last edited by schnaader; 26th May 2008 at 12:33.
    http://schnaader.info
    Damn kids. They're all alike.

  10. #10
    Member Zonder's Avatar
    Join Date
    May 2008
    Location
    Home
    Posts
    55
    Thanks
    20
    Thanked 6 Times in 5 Posts
    thx for big files support, tested on few small 200mb files - everything ok. Will let it run per night on big one.

    someone said ccm is tool of the year, only after precomp in my book
    Last edited by Zonder; 28th May 2008 at 03:24.

  11. #11
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    870
    Thanks
    47
    Thanked 105 Times in 83 Posts
    Sorry i haven't responded for the v2 test.

    my main machine upggrading is taking far more time than i first thought.
    two dead EVGA 780i boards. just got me a Gigabyte X48.

    i well test soon though

  12. #12
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    870
    Thanks
    47
    Thanked 105 Times in 83 Posts
    it stills chokes up on some files



    Code:
    E:\TESTTMP>precomp.exe -r -v I_AM_LEGEND.pcf
    
    Precomp v0.3.8 - > 2GB v2 DEVELOPMENT version - NO OFFICIAL RELEASE!
    Free for non-commercial use - Copyright 2006,2007 by Christian Schneider
    
    Input file: I_AM_LEGEND.pcf
    Output file: i_am_legend.iso
    
    Using PACKJPG.DLL for JPG recompression.
    
    --> packJPG DLL v2.3b (20/12/2007) by Matthias Stirner <--
    More about PackJPG here: http://www.elektronik.htw-aalen.de/packjpg
    
    Uncompressed data, length=961012651
    Decompressed data - GZip
    Compression level: 1
    Window size: 15
    Memory level: 1
    GZip header length: 60
    Recompressed Length: 11 - Decompressed length: 139
    Uncompressed data, length=ERROR: print64 called with negative value
    
    
    ERROR: There is not enough space on disk
    
    E:\TESTTMP>
    note that i did only run the fast mode to create the pcf file

    i will try to see if a normale mode created pcf generates same error

  13. #13
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    551
    Thanks
    206
    Thanked 182 Times in 87 Posts
    Quote Originally Posted by SvenBent View Post
    it stills chokes up on some files

    Code:
    E:\TESTTMP>precomp.exe -r -v I_AM_LEGEND.pcf
    
    (...)
    
    Uncompressed data, length=ERROR: print64 called with negative value
    
    
    ERROR: There is not enough space on disk
    
    E:\TESTTMP>
    note that i did only run the fast mode to create the pcf file

    i will try to see if a normale mode created pcf generates same error
    Thanks for testing! This bug is caused by a uncompressed stream that is larger than 2 GB - the value is stored correctly, but the recompression routines read it wrong.

    I fixed this and uploaded a new version (v3):

    New test version (v3): http://schnaader.info/precomp038_2gb_test_v3.zip

    It is compatible to v2, so you can recompress archives you made using v2 with it.
    http://schnaader.info
    Damn kids. They're all alike.

  14. #14
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    870
    Thanks
    47
    Thanked 105 Times in 83 Posts
    just got V3 will test it right now

    just to clarify when you say recompress. You men restore the original file right ?

  15. #15
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    870
    Thanks
    47
    Thanked 105 Times in 83 Posts
    The V3 restored the .pcf file back to an .iso without errors.
    and the md5 checksum is the same as the original.


    going to test at handful of big files tomorrow

  16. #16
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    551
    Thanks
    206
    Thanked 182 Times in 87 Posts
    Quote Originally Posted by SvenBent View Post
    just to clarify when you say recompress. You men restore the original file right ?
    Exactly It's like this:

    Original -> precompress with Precomp -> PCF
    PCF -> compress with your favorite archiver/compressor -> XXX

    XXX -> decompress with your favorite archiver/compressor -> PCF
    PCF -> recompress with Precomp -> Original

    Perhaps I'll find time to create some images for this and other explanatory things and put them on the site, because new users of Precomp often get confused.
    http://schnaader.info
    Damn kids. They're all alike.

  17. #17
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    870
    Thanks
    47
    Thanked 105 Times in 83 Posts
    I just tried with 3 dvd iso files
    5.68GB 4.86GB 4.35GB

    after precompresssiong and restoring, all where identical by md5 checking with the originale file

    i also noticed the they where all suggesting to use this -c14 -m1 for this file.
    maybe that holds true for all dvd iso files.

    i will try to make some gigatic pdf file with jpgs inside for futher testing.


    Oh and BTW
    precompressing three files in the same directory work without errors

  18. #18
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    551
    Thanks
    206
    Thanked 182 Times in 87 Posts
    Quote Originally Posted by SvenBent View Post
    I just tried with 3 dvd iso files
    5.68GB 4.86GB 4.35GB

    after precompresssiong and restoring, all where identical by md5 checking with the originale file
    That sounds good. At the moment I'm improving some other code, but it seems that there'll be a final 0.3.8 release soon.

    Quote Originally Posted by SvenBent View Post
    i also noticed the they where all suggesting to use this -c14 -m1 for this file.
    maybe that holds true for all dvd iso files.
    I don't think so, because ISO files are containers and the stream types used depend on the content. However, there could be something in the ISO header that Precomp mistakes for a GZip header or something like this.

    Again, thanks for testing!
    http://schnaader.info
    Damn kids. They're all alike.

  19. #19
    Member
    Join Date
    Jul 2008
    Posts
    54
    Thanks
    0
    Thanked 0 Times in 0 Posts
    I'm a compression data fan, precomp it's really amazing i tried this version using a knoppix 5.1 CD live version (it use zlib data)
    I unpacked and repacked and works greats!!

    What about to add:
    -LZMA/7-zip
    -Improve png detection
    -Huffman metohd
    -LZO
    -LZX, LZS and or LZW
    I know it's to much, but it works great...
    Maybe open the code can helps to improve more fast and get more ideas
    Thanks!

  20. #20
    Member
    Join Date
    Jan 2007
    Location
    Moscow
    Posts
    239
    Thanks
    0
    Thanked 3 Times in 1 Post
    Thanks to author for the useful program! I'd like to see in future:
    1. UUE\Base64\etc. support, since usually i save interesting web pages in .mht format for later reading. All binary and, sometimes text files in this archive format are encoded;
    2. Verbose option, when precomp will show offsets and lengths of decoded data chunks;
    3. Ability to save decoded data separately chunk by chunk (both encoded and decoded) with folder-filtering by data type (pdf, gzip, png, etc).
    Thanks!

  21. #21
    Member
    Join Date
    May 2008
    Location
    Earth
    Posts
    115
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by nimdamsk View Post
    2. Verbose option, when precomp will show offsets and lengths of decoded data chunks;
    Try -v switch

  22. #22
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    551
    Thanks
    206
    Thanked 182 Times in 87 Posts
    Quote Originally Posted by John View Post
    I'm a compression data fan, precomp it's really amazing i tried this version using a knoppix 5.1 CD live version (it use zlib data)
    I unpacked and repacked and works greats!!

    What about to add:
    -LZMA/7-zip
    -Improve png detection
    -Huffman metohd
    -LZO
    -LZX, LZS and or LZW
    I know it's to much, but it works great...
    Maybe open the code can helps to improve more fast and get more ideas
    Thanks!
    Yes, releasing the source code will be useful. I plan to get the source clean-up and all the main features (like multiple file support etc.) done until version 0.5 and then release the source code, this will speed up the work on LZ* and other additional algorithms.

    Quote Originally Posted by nimdamsk
    Thanks to author for the useful program! I'd like to see in future:
    1. UUE\Base64\etc. support, since usually i save interesting web pages in .mht format for later reading. All binary and, sometimes text files in this archive format are encoded;
    Base64/MIME support will be in the final version 0.3.8 or in the next version. I originally wanted to use it for mail databases (Thunderbird, Outlook), but I'll also have a look at .mht.

    Quote Originally Posted by nimdamsk
    3. Ability to save decoded data separately chunk by chunk (both encoded and decoded) with folder-filtering by data type (pdf, gzip, png, etc).
    What exactly do you mean? Do you want the chunks sorted for better compression?
    http://schnaader.info
    Damn kids. They're all alike.

  23. #23
    Member
    Join Date
    Jan 2007
    Location
    Moscow
    Posts
    239
    Thanks
    0
    Thanked 3 Times in 1 Post
    Quote Originally Posted by schnaader View Post
    What exactly do you mean? Do you want the chunks sorted for better compression?
    I mean i'd like to use Precomp like a ripper When analysing different firmware files which as often as not are compressed (with lzma or zlib) i'd like Precomp to rip compressed chunks, decompress and save them both. Well, i'll sort extracted files myself

Similar Threads

  1. Precomp 0.4
    By schnaader in forum Data Compression
    Replies: 190
    Last Post: 5th October 2010, 15:13
  2. Does rep support >2GB Dictionary size ?
    By SvenBent in forum Data Compression
    Replies: 12
    Last Post: 6th June 2009, 00:08
  3. Precomp (and Precomp Comfort) in 315 kb
    By Yuri Grille. in forum Data Compression
    Replies: 2
    Last Post: 1st April 2009, 19:40
  4. Precomp 0.3.5 is out!
    By squxe in forum Forum Archive
    Replies: 1
    Last Post: 20th August 2007, 14:55
  5. Precomp 0.3.3 is out!
    By squxe in forum Forum Archive
    Replies: 1
    Last Post: 20th July 2007, 17:27

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •