Precomp v0.3.8 >2GB test version

[schnaader]

I just managed to compile a test version of Precomp that could be able to process files > 2 GB, it would be nice if someone could check this, especially for files even greater than 4 GB.

Get the test version here: http://schnaader.info/precomp038_2gb_test_v3.zip
(Edit: Changed link to new v2 version, also see thread below)
(Edit2: v3 is out )

Use only for testing, I'm pretty sure there'll be some bugs in this test version!

[SvenBent]

Since i was the one to requeste/mention it public. i will gladly do some >2gb and >4gb file test.

going to start in about now...

[SvenBent]

Since you are still using ~tempX.dat files i guess that i still can't use it on several files in the same directory ?.

cant we make the files kinda random. or use the same name as the intput file ?

it would really sped up my compression/decompression as i would be able to utilize more cores.

[SvenBent]

i just used precomp on a 408gb .zip file.
the file was made with 7-zip fast and had the contents o three copies of my warcraft3 folder and somde pdf documents.
the compression raita was around 95%

the originale file and the precomped-restored file had tge same md5 checksum.


but here is my concerne., alle the three files was all 4.08gb in size

test.pcf - 4387695058
test.zip.restored - 4387673979
test.zip.org.original - 4387673979

the PCF file only grew with 21079 bytes... i had thougth that the .pcf file would grow back to something like size of the uncompressed data.


I'm going to see what kind of better compression the pcf would give me compared to the originale zip

[schnaader]

Since you are still using ~tempX.dat files i guess that i still can't use it on several files in the same directory ?.

cant we make the files kinda random. or use the same name as the intput file ?

it would really sped up my compression/decompression as i would be able to utilize more cores.

OK, I think "same name as the input file" sounds good, I'll add this.

i just used precomp on a 408gb .zip file.
the file was made with 7-zip fast and had the contents o three copies of my warcraft3 folder and somde pdf documents.
the compression raita was around 95%

the originale file and the precomped-restored file had tge same md5 checksum.

...... snip ......

the PCF file only grew with 21079 bytes... i had thougth that the .pcf file would grow back to something like size of the uncompressed data.


I'm going to see what kind of better compression the pcf would give me compared to the originale zip

I guess only some of the .ZIP contents could be recompressed, since Precomp performs bad on big ZIP streams (> 500 KB or so). Compression will not be much better, I guess. The next version will perhaps feature multiple file support, so then you can just precompress the original data without creating a ZIP before.
However, the MD5 checksum is the same, that sounds great. Thanks a lot for testing!

[SvenBent]

just to beat you and advance.

when doing multiple file support, could you make it SMP aware? so that it would work on the same numbers of files as cores in parallel.
That would be so cool

and to correct myself,
it was a 4.08GB file. not 408GB


going for some 6gb and 8 gb iso files next.

[SvenBent]

just made a 3.76GB zip file with 7-zip (fast)

it sucefully got precomped to a PCF file.

but when restorin the originale file made this error

C:\>precomp.exe -r "Librarian II.pcf"

Precomp v0.3.8 - > 2GB DEVELOPMENT version - NO OFFICIAL RELEASE!
Free for non-commercial use - Copyright 2006,2007 by Christian Schneider

Input file: Librarian II.pcf
Output file: librarian ii.zip

Using PACKJPG.DLL for JPG recompression.

--> packJPG DLL v2.3b (20/12/2007) by Matthias Stirner <--
More about PackJPG here: http://www.elektronik.htw-aalen.de/packjpg

 16.7%
ERROR: There is not enough space on disk

I tested the zip file with 7-zip test. and it was ok.
then i made a new pcf file.
but trying to restore the original file back the same error occurred.

i still have the zip and pcf file if you want them.



ups and BTW i had 50GB free space one the drive

[schnaader]

 16.7%
ERROR: There is not enough space on disk

I tested the zip file with 7-zip test. and it was ok.
then i made a new pcf file.
but trying to restore the original file back the same error occurred.

i still have the zip and pcf file if you want them.

ups and BTW i had 50GB free space one the drive

The best would be if you'd try to restore it again and use the -v debug switch. This would reveal which stream type led to the crash. I'll have a look at the code today and run some tests on various stream types, so I'll probably find the bug.

[schnaader]

There was a bug when processing penalty bytes that led to file positions that were very big, so that's why "Not enough space on disk" was shown.

I fixed the bug and uploaded a new test version. "v2" was added both to the archive name and the program message to be able to keep both versions apart, the old test version isn't available anymore.

New test version (v2): http://schnaader.info/precomp038_2gb_test_v2.zip

(Edit: Just reuploaded a version that creates unique temporary files, so running multiple Precomp tasks is now possible even in the same directory)

[Zonder]

thx for big files support, tested on few small 200mb files - everything ok. Will let it run per night on big one.

someone said ccm is tool of the year, only after precomp in my book

[SvenBent]

Sorry i haven't responded for the v2 test.

my main machine upggrading is taking far more time than i first thought.
two dead EVGA 780i boards. just got me a Gigabyte X48.

i well test soon though

[SvenBent]

it stills chokes up on some files

E:\TESTTMP>precomp.exe -r -v I_AM_LEGEND.pcf

Precomp v0.3.8 - > 2GB v2 DEVELOPMENT version - NO OFFICIAL RELEASE!
Free for non-commercial use - Copyright 2006,2007 by Christian Schneider

Input file: I_AM_LEGEND.pcf
Output file: i_am_legend.iso

Using PACKJPG.DLL for JPG recompression.

--> packJPG DLL v2.3b (20/12/2007) by Matthias Stirner <--
More about PackJPG here: http://www.elektronik.htw-aalen.de/packjpg

Uncompressed data, length=961012651
Decompressed data - GZip
Compression level: 1
Window size: 15
Memory level: 1
GZip header length: 60
Recompressed Length: 11 - Decompressed length: 139
Uncompressed data, length=ERROR: print64 called with negative value


ERROR: There is not enough space on disk

E:\TESTTMP>

note that i did only run the fast mode to create the pcf file

i will try to see if a normale mode created pcf generates same error

[schnaader]

it stills chokes up on some files

E:\TESTTMP>precomp.exe -r -v I_AM_LEGEND.pcf

(...)

Uncompressed data, length=ERROR: print64 called with negative value


ERROR: There is not enough space on disk

E:\TESTTMP>

note that i did only run the fast mode to create the pcf file

i will try to see if a normale mode created pcf generates same error

Thanks for testing! This bug is caused by a uncompressed stream that is larger than 2 GB - the value is stored correctly, but the recompression routines read it wrong.

I fixed this and uploaded a new version (v3):

New test version (v3): http://schnaader.info/precomp038_2gb_test_v3.zip

It is compatible to v2, so you can recompress archives you made using v2 with it.

[SvenBent]

just got V3 will test it right now

just to clarify when you say recompress. You men restore the original file right ?

[SvenBent]

The V3 restored the .pcf file back to an .iso without errors.
and the md5 checksum is the same as the original.


going to test at handful of big files tomorrow

[schnaader]

just to clarify when you say recompress. You men restore the original file right ?

Exactly It's like this:

Original -> precompress with Precomp -> PCF
PCF -> compress with your favorite archiver/compressor -> XXX

XXX -> decompress with your favorite archiver/compressor -> PCF
PCF -> recompress with Precomp -> Original

Perhaps I'll find time to create some images for this and other explanatory things and put them on the site, because new users of Precomp often get confused.

[SvenBent]

I just tried with 3 dvd iso files
5.68GB 4.86GB 4.35GB

after precompresssiong and restoring, all where identical by md5 checking with the originale file

i also noticed the they where all suggesting to use this -c14 -m1 for this file.
maybe that holds true for all dvd iso files.

i will try to make some gigatic pdf file with jpgs inside for futher testing.


Oh and BTW
precompressing three files in the same directory work without errors

[schnaader]

I just tried with 3 dvd iso files
5.68GB 4.86GB 4.35GB

after precompresssiong and restoring, all where identical by md5 checking with the originale file

That sounds good. At the moment I'm improving some other code, but it seems that there'll be a final 0.3.8 release soon.

i also noticed the they where all suggesting to use this -c14 -m1 for this file.
maybe that holds true for all dvd iso files.

I don't think so, because ISO files are containers and the stream types used depend on the content. However, there could be something in the ISO header that Precomp mistakes for a GZip header or something like this.

Again, thanks for testing!

[John]

I'm a compression data fan, precomp it's really amazing i tried this version using a knoppix 5.1 CD live version (it use zlib data)
I unpacked and repacked and works greats!!

What about to add:
-LZMA/7-zip
-Improve png detection
-Huffman metohd
-LZO
-LZX, LZS and or LZW
I know it's to much, but it works great...
Maybe open the code can helps to improve more fast and get more ideas
Thanks!

[nimdamsk]

Thanks to author for the useful program! I'd like to see in future:
1. UUE\Base64\etc. support, since usually i save interesting web pages in .mht format for later reading. All binary and, sometimes text files in this archive format are encoded;
2. Verbose option, when precomp will show offsets and lengths of decoded data chunks;
3. Ability to save decoded data separately chunk by chunk (both encoded and decoded) with folder-filtering by data type (pdf, gzip, png, etc).
Thanks!

[IsName]

2. Verbose option, when precomp will show offsets and lengths of decoded data chunks;

Try -v switch

[schnaader]

I'm a compression data fan, precomp it's really amazing i tried this version using a knoppix 5.1 CD live version (it use zlib data)
I unpacked and repacked and works greats!!

What about to add:
-LZMA/7-zip
-Improve png detection
-Huffman metohd
-LZO
-LZX, LZS and or LZW
I know it's to much, but it works great...
Maybe open the code can helps to improve more fast and get more ideas
Thanks!

Yes, releasing the source code will be useful. I plan to get the source clean-up and all the main features (like multiple file support etc.) done until version 0.5 and then release the source code, this will speed up the work on LZ* and other additional algorithms.

Thanks to author for the useful program! I'd like to see in future:
1. UUE\Base64\etc. support, since usually i save interesting web pages in .mht format for later reading. All binary and, sometimes text files in this archive format are encoded;

Base64/MIME support will be in the final version 0.3.8 or in the next version. I originally wanted to use it for mail databases (Thunderbird, Outlook), but I'll also have a look at .mht.

3. Ability to save decoded data separately chunk by chunk (both encoded and decoded) with folder-filtering by data type (pdf, gzip, png, etc).

What exactly do you mean? Do you want the chunks sorted for better compression?

[nimdamsk]

What exactly do you mean? Do you want the chunks sorted for better compression?

I mean i'd like to use Precomp like a ripper When analysing different firmware files which as often as not are compressed (with lzma or zlib) i'd like Precomp to rip compressed chunks, decompress and save them both. Well, i'll sort extracted files myself