zpaq updates

[Matt Mahoney]

Nice improvement. Some results for bmp_j1 on the Kodak set.

zpaq108 cmax.cfg
 503,436 kodim01.bmp.zpaq (9 sec per file optimized)
 378,583 kodim02.bmp.zpaq
 327,005 kodim03.bmp.zpaq
 441,954 kodim04.bmp.zpaq
 640,768 kodim05.bmp.zpaq
 458,957 kodim06.bmp.zpaq
 385,302 kodim07.bmp.zpaq
 665,842 kodim08.bmp.zpaq
 392,373 kodim09.bmp.zpaq
 428,791 kodim10.bmp.zpaq
 465,125 kodim11.bmp.zpaq
 359,799 kodim12.bmp.zpaq
 649,429 kodim13.bmp.zpaq
 535,099 kodim14.bmp.zpaq
 427,761 kodim15.bmp.zpaq
 359,518 kodim16.bmp.zpaq
 420,052 kodim17.bmp.zpaq
 633,543 kodim18.bmp.zpaq
 472,418 kodim19.bmp.zpaq
 352,003 kodim20.bmp.zpaq
 470,359 kodim21.bmp.zpaq
 579,559 kodim22.bmp.zpaq
 464,091 kodim23.bmp.zpaq
 546,375 kodim24.bmp.zpaq
11,358,142 bytes

zpaq108 cbmp_j1.cfg
 395,186 kodim01.bmp.zpaq (8 sec per file optimized)
 320,798 kodim02.bmp.zpaq
 268,654 kodim03.bmp.zpaq
 348,623 kodim04.bmp.zpaq
 470,393 kodim05.bmp.zpaq
 366,516 kodim06.bmp.zpaq
 305,855 kodim07.bmp.zpaq
 462,274 kodim08.bmp.zpaq
 314,971 kodim09.bmp.zpaq
 331,017 kodim10.bmp.zpaq
 355,667 kodim11.bmp.zpaq
 299,517 kodim12.bmp.zpaq
 501,523 kodim13.bmp.zpaq
 420,000 kodim14.bmp.zpaq
 332,963 kodim15.bmp.zpaq
 301,618 kodim16.bmp.zpaq
 323,406 kodim17.bmp.zpaq
 475,853 kodim18.bmp.zpaq
 373,410 kodim19.bmp.zpaq
 287,555 kodim20.bmp.zpaq
 375,628 kodim21.bmp.zpaq
 432,041 kodim22.bmp.zpaq
 338,430 kodim23.bmp.zpaq
 393,155 kodim24.bmp.zpaq
8,795,053 bytes

paq8px_v60_turbo -6
 379,021 kodim01.bmp.paq8px (no color transform)
 311,804 kodim02.bmp.paq8px
 251,387 kodim03.bmp.paq8px
 327,269 kodim04.bmp.paq8px
 438,515 kodim05.bmp.paq8px
 349,815 kodim06.bmp.paq8px
 283,934 kodim07.bmp.paq8px
 433,936 kodim08.bmp.paq8px
 300,962 kodim09.bmp.paq8px
 310,515 kodim10.bmp.paq8px
 338,414 kodim11.bmp.paq8px
 281,905 kodim12.bmp.paq8px
 475,658 kodim13.bmp.paq8px
 394,315 kodim14.bmp.paq8px
 313,626 kodim15.bmp.paq8px
 286,418 kodim16.bmp.paq8px
 304,319 kodim17.bmp.paq8px
 444,424 kodim18.bmp.paq8px
 353,101 kodim19.bmp.paq8px
 276,842 kodim20.bmp.paq8px
 356,238 kodim21.bmp.paq8px
 405,036 kodim22.bmp.paq8px
 312,148 kodim23.bmp.paq8px
 368,399 kodim24.bmp.paq8px
8,298,001 bytes

paq8px_v64 -6
 338,569 kodim01.bmp.paq8px (25 sec per file)
 273,886 kodim02.bmp.paq8px
 230,970 kodim03.bmp.paq8px
 290,665 kodim04.bmp.paq8px
 393,021 kodim05.bmp.paq8px
 313,474 kodim06.bmp.paq8px
 256,819 kodim07.bmp.paq8px
 392,708 kodim08.bmp.paq8px
 271,395 kodim09.bmp.paq8px
 278,701 kodim10.bmp.paq8px
 303,628 kodim11.bmp.paq8px
 253,575 kodim12.bmp.paq8px
 432,576 kodim13.bmp.paq8px
 352,878 kodim14.bmp.paq8px
 283,793 kodim15.bmp.paq8px
 258,925 kodim16.bmp.paq8px
 278,275 kodim17.bmp.paq8px
 406,074 kodim18.bmp.paq8px
 319,608 kodim19.bmp.paq8px
 259,328 kodim20.bmp.paq8px
 323,466 kodim21.bmp.paq8px
 366,561 kodim22.bmp.paq8px
 284,988 kodim23.bmp.paq8px
 333,431 kodim24.bmp.paq8px
7,497,314 bytes

Compression time is about 21 sec. per file with zpaq 1.07 and 8 seconds after optimizing with zpaq 1.08. paq8px takes about 25 seconds per file. v60 does not include a color transform. v64 uses the transform (b,g,r) -> (g,g-r,g-b).

[Jan Ondrus]

bmp_j2.cfg
- improved 24-bit image compression model and added (b,g,r) -> (g,g-r,g-b) color transformation (colorpre.cfg).

rafale.bmp ... 4149414 bytes
zpaq cmax.cfg ... 714263 bytes
zpaq cbmp_j1.cfg ... 538228 bytes
zpaq cbmp_j2.cfg ... 531900 bytes

kodim01.bmp ... 1179702 bytes
zpaq cmax.cfg ... 503436 bytes
zpaq cbmp_j1.cfg ... 395186 bytes
zpaq cbmp_j2.cfg ... 340242 bytes

[Jan Ondrus]

1) I can't detect end of file in HCOMP section (it is possible in PCOMP)
2) I miss debug option for zpaq to write output of postprocessor (PCOMP) to file if comparison with unpreprocessed input FAILED.

I solved this by adding this two lines before running postprocessor.

out=fopen("xxx", "wb");
      pz.output=out;

[Matt Mahoney]

Very nice result. Compression is almost as good as paq8px_v64 and over twice as fast (11 sec. vs 25 sec. per file).

zpaq cbmp_j2.cfg
 340,242 kodim01.bmp.zpaq (11 sec per file)
 274,770 kodim02.bmp.zpaq
 233,212 kodim03.bmp.zpaq
 292,103 kodim04.bmp.zpaq
 393,855 kodim05.bmp.zpaq
 315,403 kodim06.bmp.zpaq
 257,922 kodim07.bmp.zpaq
 392,021 kodim08.bmp.zpaq
 272,943 kodim09.bmp.zpaq
 280,186 kodim10.bmp.zpaq
 304,658 kodim11.bmp.zpaq
 255,503 kodim12.bmp.zpaq
 433,220 kodim13.bmp.zpaq
 355,289 kodim14.bmp.zpaq
 283,121 kodim15.bmp.zpaq
 261,497 kodim16.bmp.zpaq
 279,832 kodim17.bmp.zpaq
 405,668 kodim18.bmp.zpaq
 321,464 kodim19.bmp.zpaq
 260,934 kodim20.bmp.zpaq
 325,393 kodim21.bmp.zpaq
 367,238 kodim22.bmp.zpaq
 286,420 kodim23.bmp.zpaq
 335,435 kodim24.bmp.zpaq
7,528,329 bytes

1. It is true you can't detect EOF in HCOMP. The model can only handle 8 bits per byte. A possible fix is to add info in the preprocessor, such as file size at beginning or special symbol at end.

2. Another way to get the output of a failed preprocessor test:

zpaq rcolorpre.cfg input tmp
zpaq prbmp_j2.cfg tmp output

A production compressor for BMP (with no config file or external preprocessor) would probably read the BMP file, get the image width, and modify the ZPAQL code appropriately. You can get a dump of the ZPAQL code as a C++ array of bytes from zpaqopt.h and use ZPAQL::read(Reader(array, size)) to load the code before compression. Before writing the header and compressing, modify the part of the code that depends on image width, and put each image in a separate block.

[Darek]

Hi!
bmp_j2 is really a best performer. My scores:

1.bmp = 600'054
1.bmp/paq8px1 -7 = 264'193
1.bmp/paq8px64 -7 = 236'363
1.bmp/zpaq1.08 max = 358'276
1.bmp/zpaq1.08 bmp_j2 = 235'691

a.tif = 2'870'552
a.tif/paq8px1 -8 = 864'257
a.tif/paq8px64 -8 = 438'372
a.tif/zpaq1.08 max = 1'005'564
a.tif/zpaq1.08 bmp_j2 = FAILED (bmp_j1 = 790'261)

b.tga = 2'872'818
b.tga/paq8px1 -8 = 813'144
b.tga/paq8px64 -8 = 402'625
b.tga/zpaq1.08 max = 945'034
b.tga/zpaq1.08 bmp_j2 = 616'520

Of course, bmp_j2 doesn't have TIFF or TGA, recognition, then scores are worse than px64, however still much better than max config or px1.

I've tried to test latest exe and bwt configs, but it need compiled bwtpre and exepre, which I haven't. Does anybody have compiled cersions of these files?

Darek

[Jan Ondrus]

I've tried to test latest exe and bwt configs, but it need compiled bwtpre and exepre, which I haven't. Does anybody have compiled cersions of these files?

This versions seems to contain compiled preprocessors:
http://mattmahoney.net/dc/exe_j1.zip
http://mattmahoney.net/dc/bwt_j3.zip

bmp_j3.cfg - improved 24-bit compression model

rafale.bmp ... 4149414 bytes
zpaq cmax.cfg ... 714263 bytes
zpaq cbmp_j1.cfg ... 538228 bytes
zpaq cbmp_j2.cfg ... 531900 bytes
zpaq cbmp_j3.cfg ... 523693 bytes

kodim01.bmp ... 1179702 bytes
zpaq cmax.cfg ... 503436 bytes
zpaq cbmp_j1.cfg ... 395186 bytes
zpaq cbmp_j2.cfg ... 340242 bytes
zpaq cbmp_j3.cfg ... 338204 bytes

Should be as good as paq8px_v64 or slightly better on average.

EDIT:
rafale.bmp with special transformation trick:

      g^=(g&4)?3:0;
      r^=(r&8)?7:0;
      b^=(b&8)?7:0;

-> 507521 bytes

[Darek]

Jan,
many thanks for compilations.
Darek

[Jan Ondrus]

Kodak set (http://r0k.us/graphics/kodak/) - result:

zpaq108 cmax.cfg ... 11358142 bytes
zpaq cbmp_j1.cfg ... 8795053 bytes
paq8px_v60       ... 8298001 bytes
zpaq cbmp_j2.cfg ... 7528329 bytes
zpaq cbmp_j3.cfg ... 7500633 bytes
paq8px_v64       ... 7497314 bytes
zpaq cbmp_j4.cfg ... 7485249 bytes

[Matt Mahoney]

Some results with zpaq 1.08, which I just uploaded to http://mattmahoney.net/dc/

zpaq108 ocbmp_j4.cfg
 338,151 kodim01.bmp.zpaq (10-11 sec each)
 272,029 kodim02.bmp.zpaq
 230,431 kodim03.bmp.zpaq
 291,064 kodim04.bmp.zpaq
 393,110 kodim05.bmp.zpaq
 314,011 kodim06.bmp.zpaq
 255,350 kodim07.bmp.zpaq
 390,582 kodim08.bmp.zpaq
 270,386 kodim09.bmp.zpaq
 278,175 kodim10.bmp.zpaq
 303,601 kodim11.bmp.zpaq
 253,364 kodim12.bmp.zpaq
 431,880 kodim13.bmp.zpaq
 353,718 kodim14.bmp.zpaq
 281,349 kodim15.bmp.zpaq
 258,687 kodim16.bmp.zpaq
 277,247 kodim17.bmp.zpaq
 404,235 kodim18.bmp.zpaq
 319,912 kodim19.bmp.zpaq
 259,391 kodim20.bmp.zpaq
 323,942 kodim21.bmp.zpaq
 365,658 kodim22.bmp.zpaq
 285,507 kodim23.bmp.zpaq
 333,469 kodim24.bmp.zpaq
7,485,249 bytes

This compares to 7,497,314 bytes and about 25 seconds per file for paq8px_v64 -6. The "o" modifier searches for optimized code (compiled COMP and ZPAQL) and creates and compiles it as needed. This adds about 2 seconds to the first file to run the g++ compiler and put the optimized version of ZPAQ in %TEMP%. The rest will find the .exe already there and run it. This compares to 26 seconds for zpaq 1.07 or if you don't use the "o" modifier in v1.08. Decompress with "ox".

To install, you need to configure zpaqmake.bat to find zpaq.cpp (or zpaq.o) and zpaq.h and call your C++ compiler with appropriate options, then put the script and zpaq.exe somewhere in your PATH. The program name will be passed as an argument. It should be compiled with -DNDEBUG -DOPT. An example is included.

Optimized programs are created by replacing the interpreters in ZPAQL::run(), Predictor:: predict() and Predictor::update() with compiled code. -DOPT causes zpaq.cpp to strip out unneeded features (like "l" and "r" commands), and to link to this code that it just wrote. When decompressing, ZPAQ calls its optimized version of itself once for each block in the archive. For compressing, it only calls itself once because ZPAQ adds one block at a time. The name of the new program is based on the SHA1 hash of the COMP, HCOMP, and PCOMP code from the config file or block header, plus the name of the preprocessor. The decompresser doesn't check the header before decompressing. It just assumes there will be no hash collisions, so there is a 2^-160 probability of failing

You can delete the contents of %TEMP% any time you want and ZPAQL will create new files as needed.

I did not test in Linux yet but I designed with it in mind. ZPAQ will look in $TMPDIR (normally /tmp) first, then $TEMP, then in the current directory. You would need to write a shell script equivalent to zpaqmake.bat. The supplied script was tested with g++, Borland and Mars compilers.

[Jan Ondrus]

bwt_slowmode1
- slow mode (using 1.25x block-size memory) inverse bwt rewritten to ZPAQL.

[Matt Mahoney]

I continue to be amazed at your work I tested on enwik9. Decompression with optimized code in BWT slow mode is faster than unoptimized code in fast mode. http://mattmahoney.net/dc/text.html#1493

                  Compression                          Compressed size      Decompresser  Total size   Time (ns/byte)
Program            Options                           enwik8      enwik9     size (zip)   enwik9+prog  Comp Decomp  Mem Alg Note
-------            -------                         ----------  -----------  -----------  -----------  ----- -----  --- --- ----
zpaq 1.07          cbwt_j2.cfg,18                  20,756,888  174,171,969     13,421 x  174,185,390   5593  4347 1443 BWT 26
zpaq 1.08          ocbwt_slowmodel_1GB_block.cfg   20,756,996  163,565,006     29,153 x  163,594,159   7957  3875 1443 BWT 26

[Jan Ondrus]

You could even write a JPEG compressor that undid the Huffman coded DCT coefficients and append a ZPAQL program to redo the Huffman coding. Any existing ZPAQ decompressor could then read your files even though the compression algorithm hadn't even been written yet

Done.
It has no special jpeg model, only transformation for decoding/coding Huffman.

a10.jpg ... 842468 bytes
zpaq.exe cjpg_test1.cfg ... 745710 bytes (-11.5%)

EDIT:
attachment removed - please download fixed version

[Matt Mahoney]

The preprocessor test failed on the 3 act-jpeg test images.
http://compression.ca/act/act-jpeg.html

[Jan Ondrus]

The preprocessor test failed on the 3 act-jpeg test images.
http://compression.ca/act/act-jpeg.html

There was bug in read_header() procedure:
(r[2] > 192 || r[2] < 255) --> (r[2] > 192 && r[2] < 255)

Here is fixed version test1 + version test2 with basic jpeg model.

DSCN3974.jpg - 1114198 bytes
zpaq cjpg_test1.cfg - 989232 bytes
zpaq cjpg_test2.cfg - 916368 bytes

[Simon Berger]

You are unbelievable!

[PiPPoNe92]

W-O-N-D-E-R-F-U-L!

[Matt Mahoney]

Unfortunately jpg_test1.cfg and jpg_test2.cfg fail on 47 of 100 files in the Picasa data set (100 random JPEGS I got from Picasa).

http://mattmahoney.net/tmp/picasa.zip

results are shown below with paq8px_v64 -6 for comparison. Files with no zpaq1 or zpaq2 compression failed the preprocessor test.

08/25/2007  04:04 PM           320,403 0607087.JPG
10/19/2009  02:58 PM           257,829 0607087.JPG.paq8px
10/19/2009  02:40 PM           286,821 0607087.JPG.zpaq1
10/19/2009  02:48 PM           269,639 0607087.JPG.zpaq2
08/25/2007  03:51 PM            26,513 070330_23310001.jpg
10/19/2009  02:58 PM            21,741 070330_23310001.jpg.paq8px
10/19/2009  02:40 PM            25,019 070330_23310001.jpg.zpaq1
10/19/2009  02:48 PM            24,210 070330_23310001.jpg.zpaq2
08/25/2007  04:03 PM           113,463 070730_Arcachon_22.jpg
10/19/2009  02:58 PM            92,361 070730_Arcachon_22.jpg.paq8px
08/25/2007  04:04 PM            85,686 070806_Main_06.jpg
10/19/2009  02:58 PM            67,946 070806_Main_06.jpg.paq8px
08/25/2007  04:03 PM         1,029,908 1010177_IMG.JPG
10/19/2009  02:59 PM           745,430 1010177_IMG.JPG.paq8px
10/19/2009  02:40 PM           895,150 1010177_IMG.JPG.zpaq1
10/19/2009  02:49 PM           798,347 1010177_IMG.JPG.zpaq2
08/25/2007  03:54 PM           601,832 110_1063.JPG
10/19/2009  02:59 PM           454,413 110_1063.JPG.paq8px
08/25/2007  03:56 PM           278,467 1180924891597.jpg
10/19/2009  02:59 PM           226,190 1180924891597.jpg.paq8px
10/19/2009  02:40 PM           254,910 1180924891597.jpg.zpaq1
10/19/2009  02:49 PM           241,356 1180924891597.jpg.zpaq2
08/25/2007  03:53 PM            60,435 1st_Haiti_Trip_012.jpg
10/19/2009  02:59 PM            50,578 1st_Haiti_Trip_012.jpg.paq8px
10/19/2009  02:40 PM            57,154 1st_Haiti_Trip_012.jpg.zpaq1
10/19/2009  02:49 PM            53,971 1st_Haiti_Trip_012.jpg.zpaq2
08/25/2007  04:03 PM           102,378 20070324_271.jpg
10/19/2009  03:00 PM            67,373 20070324_271.jpg.paq8px
10/19/2009  02:40 PM            80,213 20070324_271.jpg.zpaq1
10/19/2009  02:49 PM            81,798 20070324_271.jpg.zpaq2
08/25/2007  04:00 PM           117,272 2007398.jpg
10/19/2009  03:00 PM            82,849 2007398.jpg.paq8px
10/19/2009  02:40 PM            99,894 2007398.jpg.zpaq1
10/19/2009  02:49 PM            91,401 2007398.jpg.zpaq2
08/25/2007  04:02 PM            56,446 4215626041.jpg
10/19/2009  03:00 PM            55,432 4215626041.jpg.paq8px
08/25/2007  04:04 PM            40,595 49.jpg
10/19/2009  03:00 PM            33,494 49.jpg.paq8px
10/19/2009  02:41 PM            37,862 49.jpg.zpaq1
10/19/2009  02:49 PM            37,274 49.jpg.zpaq2
08/25/2007  03:44 PM            11,709 5.jpg
10/19/2009  03:00 PM             9,960 5.jpg.paq8px
10/19/2009  02:41 PM            12,021 5.jpg.zpaq1
10/19/2009  02:49 PM            12,167 5.jpg.zpaq2
08/25/2007  03:48 PM            47,558 53500006.jpg
10/19/2009  03:00 PM            40,601 53500006.jpg.paq8px
10/19/2009  02:41 PM            45,351 53500006.jpg.zpaq1
10/19/2009  02:49 PM            44,154 53500006.jpg.zpaq2
08/25/2007  04:05 PM           163,423 AkhaSwingFest_SIMANN_10.JPG
10/19/2009  03:00 PM           126,264 AkhaSwingFest_SIMANN_10.JPG.paq8px
08/25/2007  04:04 PM            89,780 AkhaSwingFest_SIMANN_9.JPG
10/19/2009  03:00 PM            65,877 AkhaSwingFest_SIMANN_9.JPG.paq8px
08/25/2007  03:51 PM           437,170 Alfonso_in_primo_piano.JPG
10/19/2009  03:00 PM           328,463 Alfonso_in_primo_piano.JPG.paq8px
10/19/2009  02:41 PM           378,216 Alfonso_in_primo_piano.JPG.zpaq1
10/19/2009  02:49 PM           355,727 Alfonso_in_primo_piano.JPG.zpaq2
08/25/2007  04:08 PM           262,135 ART_Fusion_13.JPG
10/19/2009  03:01 PM           205,806 ART_Fusion_13.JPG.paq8px
10/19/2009  02:41 PM           235,271 ART_Fusion_13.JPG.zpaq1
10/19/2009  02:50 PM           225,450 ART_Fusion_13.JPG.zpaq2
08/25/2007  04:07 PM           229,862 ART_Fusion_62.JPG
10/19/2009  03:01 PM           177,545 ART_Fusion_62.JPG.paq8px
10/19/2009  02:41 PM           204,539 ART_Fusion_62.JPG.zpaq1
10/19/2009  02:50 PM           196,312 ART_Fusion_62.JPG.zpaq2
08/25/2007  04:04 PM            24,369 Attachment60639.jpg
10/19/2009  03:01 PM            19,552 Attachment60639.jpg.paq8px
10/19/2009  02:41 PM            22,973 Attachment60639.jpg.zpaq1
10/19/2009  02:50 PM            22,904 Attachment60639.jpg.zpaq2
08/25/2007  04:04 PM            25,507 Attachment76429.jpg
10/19/2009  03:01 PM            20,015 Attachment76429.jpg.paq8px
10/19/2009  02:41 PM            24,237 Attachment76429.jpg.zpaq1
10/19/2009  02:50 PM            23,900 Attachment76429.jpg.zpaq2
08/25/2007  04:01 PM           317,386 august_18_2007_enso_karate_new_buffalo_beach_training_102.JPG
10/19/2009  03:01 PM           255,785 august_18_2007_enso_karate_new_buffalo_beach_training_102.JPG.paq8px
08/25/2007  03:51 PM            69,720 blogbanner.JPG
10/19/2009  03:01 PM            53,656 blogbanner.JPG.paq8px
10/19/2009  02:41 PM            61,174 blogbanner.JPG.zpaq1
10/19/2009  02:50 PM            57,957 blogbanner.JPG.zpaq2
08/25/2007  03:49 PM           310,795 buster_002.jpg
10/19/2009  03:01 PM           251,134 buster_002.jpg.paq8px
08/25/2007  03:48 PM           314,548 catnKeys200519.jpg
10/19/2009  03:02 PM           243,375 catnKeys200519.jpg.paq8px
08/25/2007  03:49 PM           232,330 catnKeys2005205.jpg
10/19/2009  03:02 PM           174,622 catnKeys2005205.jpg.paq8px
10/19/2009  02:41 PM           214,353 catnKeys2005205.jpg.zpaq1
10/19/2009  02:50 PM           201,067 catnKeys2005205.jpg.zpaq2
08/25/2007  03:47 PM           306,185 CatShotAccident.jpg
10/19/2009  03:02 PM           246,723 CatShotAccident.jpg.paq8px
10/19/2009  02:41 PM           278,183 CatShotAccident.jpg.zpaq1
10/19/2009  02:50 PM           262,670 CatShotAccident.jpg.zpaq2
08/25/2007  03:45 PM           121,575 catt_002.jpg
10/19/2009  03:02 PM            93,760 catt_002.jpg.paq8px
08/25/2007  04:04 PM            25,320 CAUDEC8F.jpg
10/19/2009  03:02 PM            20,190 CAUDEC8F.jpg.paq8px
10/19/2009  02:41 PM            24,409 CAUDEC8F.jpg.zpaq1
10/19/2009  02:50 PM            24,108 CAUDEC8F.jpg.zpaq2
08/25/2007  04:05 PM           126,693 chiangrai_068.jpg
10/19/2009  03:02 PM           100,961 chiangrai_068.jpg.paq8px
08/25/2007  04:06 PM           120,481 chiangrai_097.jpg
10/19/2009  03:02 PM            93,553 chiangrai_097.jpg.paq8px
08/25/2007  03:54 PM            29,556 CIMG04825_f.jpg
10/19/2009  03:03 PM            20,751 CIMG04825_f.jpg.paq8px
10/19/2009  02:41 PM            25,415 CIMG04825_f.jpg.zpaq1
10/19/2009  02:50 PM            24,300 CIMG04825_f.jpg.zpaq2
08/25/2007  03:54 PM           295,206 CIMG4779.JPG
10/19/2009  03:03 PM           235,010 CIMG4779.JPG.paq8px
08/25/2007  04:01 PM           207,380 Cold_Sunset.JPG
10/19/2009  03:03 PM           159,782 Cold_Sunset.JPG.paq8px
08/25/2007  03:46 PM            16,845 Copy_253_of_EXIF_Fuji1500_400x315_17K.JPG
10/19/2009  03:03 PM            13,459 Copy_253_of_EXIF_Fuji1500_400x315_17K.JPG.paq8px
10/19/2009  02:41 PM            15,706 Copy_253_of_EXIF_Fuji1500_400x315_17K.JPG.zpaq1
10/19/2009  02:50 PM            16,308 Copy_253_of_EXIF_Fuji1500_400x315_17K.JPG.zpaq2
08/25/2007  04:03 PM            41,013 co_022.jpg
10/19/2009  03:03 PM            31,623 co_022.jpg.paq8px
08/25/2007  04:06 PM           100,298 dad.jpg
10/19/2009  03:03 PM            74,638 dad.jpg.paq8px
08/25/2007  04:01 PM           285,292 day_at_the_beach_034.JPG
10/19/2009  03:03 PM           222,527 day_at_the_beach_034.JPG.paq8px
10/19/2009  02:41 PM           257,616 day_at_the_beach_034.JPG.zpaq1
10/19/2009  02:50 PM           241,047 day_at_the_beach_034.JPG.zpaq2
08/25/2007  03:44 PM            37,440 dfl_0060.jpg
10/19/2009  03:03 PM            31,515 dfl_0060.jpg.paq8px
10/19/2009  02:41 PM            35,985 dfl_0060.jpg.zpaq1
10/19/2009  02:50 PM            35,091 dfl_0060.jpg.zpaq2
08/25/2007  03:43 PM            59,016 dfl_0505.jpg
10/19/2009  03:03 PM            50,343 dfl_0505.jpg.paq8px
10/19/2009  02:42 PM            55,627 dfl_0505.jpg.zpaq1
10/19/2009  02:50 PM            53,893 dfl_0505.jpg.zpaq2
08/25/2007  04:05 PM           624,114 DilsbergNeckarsteinach_065.JPG
10/19/2009  03:04 PM           514,435 DilsbergNeckarsteinach_065.JPG.paq8px
10/19/2009  02:42 PM           563,391 DilsbergNeckarsteinach_065.JPG.zpaq1
10/19/2009  02:51 PM           529,904 DilsbergNeckarsteinach_065.JPG.zpaq2
08/25/2007  04:01 PM           387,635 DSC02753.JPG
10/19/2009  03:04 PM           315,064 DSC02753.JPG.paq8px
08/25/2007  04:00 PM           272,931 DSC02837.JPG
10/19/2009  03:04 PM           222,984 DSC02837.JPG.paq8px
08/25/2007  04:04 PM            36,819 DSC06086.jpg
10/19/2009  03:04 PM            28,830 DSC06086.jpg.paq8px
10/19/2009  02:42 PM            32,692 DSC06086.jpg.zpaq1
10/19/2009  02:51 PM            32,591 DSC06086.jpg.zpaq2
08/25/2007  03:51 PM           134,143 DSCF1348.jpg
10/19/2009  03:04 PM           110,607 DSCF1348.jpg.paq8px
10/19/2009  02:42 PM           124,180 DSCF1348.jpg.zpaq1
10/19/2009  02:51 PM           118,629 DSCF1348.jpg.zpaq2
08/25/2007  04:00 PM            46,673 DSCF1980.jpg
10/19/2009  03:04 PM            39,182 DSCF1980.jpg.paq8px
10/19/2009  02:42 PM            43,724 DSCF1980.jpg.zpaq1
10/19/2009  02:51 PM            42,173 DSCF1980.jpg.zpaq2
08/25/2007  03:58 PM           710,719 DSCN0230.JPG
10/19/2009  03:05 PM           564,387 DSCN0230.JPG.paq8px
10/19/2009  02:42 PM           628,455 DSCN0230.JPG.zpaq1
10/19/2009  02:51 PM           588,323 DSCN0230.JPG.zpaq2
08/25/2007  04:08 PM           353,399 DSCN0400.JPG
10/19/2009  03:05 PM           285,487 DSCN0400.JPG.paq8px
08/25/2007  03:55 PM           819,747 DSCN1654.JPG
10/19/2009  03:06 PM           618,875 DSCN1654.JPG.paq8px
10/19/2009  02:42 PM           736,550 DSCN1654.JPG.zpaq1
10/19/2009  02:52 PM           682,288 DSCN1654.JPG.zpaq2
08/25/2007  03:57 PM            46,556 DSCN1681.JPG
10/19/2009  03:06 PM            38,941 DSCN1681.JPG.paq8px
10/19/2009  02:43 PM            44,254 DSCN1681.JPG.zpaq1
10/19/2009  02:52 PM            42,236 DSCN1681.JPG.zpaq2
08/25/2007  04:04 PM           576,480 DSCN1937.JPG
10/19/2009  03:06 PM           416,984 DSCN1937.JPG.paq8px
08/25/2007  03:57 PM            46,182 DSCN1948.JPG
10/19/2009  03:06 PM            37,957 DSCN1948.JPG.paq8px
10/19/2009  02:43 PM            42,948 DSCN1948.JPG.zpaq1
10/19/2009  02:52 PM            41,267 DSCN1948.JPG.zpaq2
08/25/2007  03:57 PM           891,520 DSCN2865.JPG
10/19/2009  03:07 PM           674,004 DSCN2865.JPG.paq8px
08/25/2007  04:05 PM           231,252 DSC_0020.JPG
10/19/2009  03:07 PM           185,419 DSC_0020.JPG.paq8px
08/25/2007  04:06 PM           143,762 DSC_1825.jpg
10/19/2009  03:07 PM           104,474 DSC_1825.jpg.paq8px
08/25/2007  03:48 PM           666,543 DSC_3717.jpg
10/19/2009  03:08 PM           553,880 DSC_3717.jpg.paq8px
10/19/2009  02:43 PM           602,551 DSC_3717.jpg.zpaq1
10/19/2009  02:52 PM           573,262 DSC_3717.jpg.zpaq2
08/25/2007  04:07 PM           152,060 EZEGNAX21.jpg
10/19/2009  03:08 PM           144,950 EZEGNAX21.jpg.paq8px
08/25/2007  03:51 PM           222,128 grey_002.jpg
10/19/2009  03:08 PM           176,579 grey_002.jpg.paq8px
08/25/2007  04:07 PM         1,369,536 Hassam_GalleryPlayer.jpg
10/19/2009  03:09 PM         1,187,962 Hassam_GalleryPlayer.jpg.paq8px
08/25/2007  04:01 PM           218,834 HPIM2867.JPG
10/19/2009  03:09 PM           169,804 HPIM2867.JPG.paq8px
08/25/2007  04:08 PM            33,726 ILC_056.jpg
10/19/2009  03:09 PM            25,756 ILC_056.jpg.paq8px
08/25/2007  04:08 PM            40,811 ILC_072.jpg
10/19/2009  03:10 PM            30,481 ILC_072.jpg.paq8px
08/25/2007  03:54 PM           124,864 Image00006.jpg
10/19/2009  03:10 PM            81,835 Image00006.jpg.paq8px
08/25/2007  04:07 PM            45,322 image005.jpg
10/19/2009  03:10 PM            37,459 image005.jpg.paq8px
10/19/2009  02:43 PM            42,625 image005.jpg.zpaq1
10/19/2009  02:52 PM            41,343 image005.jpg.zpaq2
08/25/2007  03:52 PM         2,230,194 IMG_0010.JPG
10/19/2009  03:11 PM         1,683,525 IMG_0010.JPG.paq8px
08/25/2007  04:02 PM           212,403 IMG_0189.JPG
10/19/2009  03:11 PM           168,631 IMG_0189.JPG.paq8px
10/19/2009  02:43 PM           197,069 IMG_0189.JPG.zpaq1
10/19/2009  02:52 PM           187,237 IMG_0189.JPG.zpaq2
08/25/2007  04:03 PM            44,643 IMG_0349.JPG
10/19/2009  03:11 PM            37,010 IMG_0349.JPG.paq8px
10/19/2009  02:43 PM            42,796 IMG_0349.JPG.zpaq1
10/19/2009  02:53 PM            41,276 IMG_0349.JPG.zpaq2
08/25/2007  04:05 PM           288,511 IMG_1296.JPG
10/19/2009  03:12 PM           225,612 IMG_1296.JPG.paq8px
10/19/2009  02:43 PM           259,762 IMG_1296.JPG.zpaq1
10/19/2009  02:53 PM           247,793 IMG_1296.JPG.zpaq2
08/25/2007  03:45 PM            22,216 IMG_2549.jpg
10/19/2009  03:12 PM            18,349 IMG_2549.jpg.paq8px
10/19/2009  02:43 PM            21,659 IMG_2549.jpg.zpaq1
10/19/2009  02:53 PM            21,287 IMG_2549.jpg.zpaq2
08/25/2007  03:58 PM           139,946 IMG_3766.JPG
10/19/2009  03:12 PM           113,355 IMG_3766.JPG.paq8px
10/19/2009  02:43 PM           127,832 IMG_3766.JPG.zpaq1
10/19/2009  02:53 PM           121,987 IMG_3766.JPG.zpaq2
08/25/2007  03:51 PM           239,908 IMG_3969.JPG
10/19/2009  03:12 PM           189,964 IMG_3969.JPG.paq8px
08/25/2007  04:03 PM            82,320 JamaicaCraft_04.jpg
10/19/2009  03:12 PM            67,984 JamaicaCraft_04.jpg.paq8px
10/19/2009  02:43 PM            76,252 JamaicaCraft_04.jpg.zpaq1
10/19/2009  02:53 PM            73,404 JamaicaCraft_04.jpg.zpaq2
08/25/2007  04:05 PM            35,833 jugs424px.jpg
10/19/2009  03:12 PM            28,976 jugs424px.jpg.paq8px
10/19/2009  02:43 PM            35,232 jugs424px.jpg.zpaq1
10/19/2009  02:53 PM            32,761 jugs424px.jpg.zpaq2
08/25/2007  03:48 PM           199,101 kittycat_015.JPG
10/19/2009  03:12 PM           159,608 kittycat_015.JPG.paq8px
10/19/2009  02:44 PM           180,039 kittycat_015.JPG.zpaq1
10/19/2009  02:53 PM           172,243 kittycat_015.JPG.zpaq2
08/25/2007  04:05 PM           207,893 kohphangan4_014.jpg
10/19/2009  03:12 PM           170,150 kohphangan4_014.jpg.paq8px
08/25/2007  04:05 PM           105,874 kohphangan_missingshots_029.jpg
10/19/2009  03:12 PM            81,357 kohphangan_missingshots_029.jpg.paq8px
08/25/2007  04:06 PM           119,408 kohphangan_missingshots_033.jpg
10/19/2009  03:13 PM            92,913 kohphangan_missingshots_033.jpg.paq8px
08/25/2007  04:01 PM         2,065,498 March_2007_014.jpg
10/19/2009  03:14 PM         1,673,043 March_2007_014.jpg.paq8px
08/25/2007  04:01 PM            19,571 Marian_skotti.jpg
10/19/2009  03:14 PM            15,403 Marian_skotti.jpg.paq8px
10/19/2009  02:44 PM            18,751 Marian_skotti.jpg.zpaq1
10/19/2009  02:53 PM            17,898 Marian_skotti.jpg.zpaq2
08/25/2007  03:54 PM           374,250 mountains_july_2007_022.JPG
10/19/2009  03:14 PM           303,937 mountains_july_2007_022.JPG.paq8px
08/25/2007  03:46 PM            41,114 Mu_007.jpg
10/19/2009  03:14 PM            31,768 Mu_007.jpg.paq8px
08/25/2007  04:06 PM           239,129 Neridas_21st_015.jpg
10/19/2009  03:14 PM           195,304 Neridas_21st_015.jpg.paq8px
08/25/2007  04:07 PM           172,078 Newtown_street_art_2.jpg
10/19/2009  03:14 PM           125,609 Newtown_street_art_2.jpg.paq8px
10/19/2009  02:44 PM           154,540 Newtown_street_art_2.jpg.zpaq1
10/19/2009  02:53 PM           136,108 Newtown_street_art_2.jpg.zpaq2
08/25/2007  04:05 PM           123,938 newyork2007_15.jpg
10/19/2009  03:15 PM            98,128 newyork2007_15.jpg.paq8px
08/25/2007  04:00 PM           359,270 OB_62607_019a.jpg
10/19/2009  03:15 PM           290,547 OB_62607_019a.jpg.paq8px
10/19/2009  02:44 PM           325,536 OB_62607_019a.jpg.zpaq1
10/19/2009  02:53 PM           307,037 OB_62607_019a.jpg.zpaq2
08/25/2007  04:06 PM           162,467 Overworked.jpg
10/19/2009  03:15 PM           119,451 Overworked.jpg.paq8px
08/25/2007  04:01 PM            35,519 P1090197.JPG
10/19/2009  03:15 PM            29,422 P1090197.JPG.paq8px
10/19/2009  02:44 PM            34,167 P1090197.JPG.zpaq1
10/19/2009  02:53 PM            32,385 P1090197.JPG.zpaq2
08/25/2007  03:56 PM           182,985 PICT0947.JPG
10/19/2009  03:15 PM           142,534 PICT0947.JPG.paq8px
08/25/2007  03:58 PM           912,241 Reflection_GalleryPlayer.jpg
10/19/2009  03:16 PM           720,418 Reflection_GalleryPlayer.jpg.paq8px
08/25/2007  04:07 PM           124,076 Scene_from_the_Book_of_Portals_in_the_Tomb_of_Ramesses_I.jpg
10/19/2009  03:16 PM            99,982 Scene_from_the_Book_of_Portals_in_the_Tomb_of_Ramesses_I.jpg.paq8px
10/19/2009  02:44 PM           115,396 Scene_from_the_Book_of_Portals_in_the_Tomb_of_Ramesses_I.jpg.zpaq1
10/19/2009  02:53 PM           110,654 Scene_from_the_Book_of_Portals_in_the_Tomb_of_Ramesses_I.jpg.zpaq2
08/25/2007  03:46 PM           125,075 shadz_017.jpg
10/19/2009  03:16 PM            94,594 shadz_017.jpg.paq8px
08/25/2007  04:06 PM            11,927 ShowLetterj.jpg
10/19/2009  03:16 PM             9,547 ShowLetterj.jpg.paq8px
10/19/2009  02:44 PM            11,909 ShowLetterj.jpg.zpaq1
10/19/2009  02:53 PM            12,168 ShowLetterj.jpg.zpaq2
08/25/2007  03:53 PM            20,317 SUN04MB039001[2].jpg
10/19/2009  03:16 PM            16,022 SUN04MB039001[2].jpg.paq8px
10/19/2009  02:44 PM            19,534 SUN04MB039001[2].jpg.zpaq1
10/19/2009  02:53 PM            18,966 SUN04MB039001[2].jpg.zpaq2
08/25/2007  04:03 PM            42,781 Sunset_16.jpg
10/19/2009  03:16 PM            33,764 Sunset_16.jpg.paq8px
10/19/2009  02:44 PM            41,466 Sunset_16.jpg.zpaq1
10/19/2009  02:54 PM            37,664 Sunset_16.jpg.zpaq2
08/25/2007  03:57 PM            42,926 Terry_disturbing_tthe_peace_on_Beinn_na_Lap.JPG
10/19/2009  03:16 PM            35,314 Terry_disturbing_tthe_peace_on_Beinn_na_Lap.JPG.paq8px
10/19/2009  02:44 PM            39,973 Terry_disturbing_tthe_peace_on_Beinn_na_Lap.JPG.zpaq1
10/19/2009  02:54 PM            39,012 Terry_disturbing_tthe_peace_on_Beinn_na_Lap.JPG.zpaq2
08/25/2007  04:08 PM             2,852 The_Letter.jpg
10/19/2009  03:16 PM             2,579 The_Letter.jpg.paq8px
10/19/2009  02:44 PM             3,783 The_Letter.jpg.zpaq1
10/19/2009  02:54 PM             4,293 The_Letter.jpg.zpaq2
08/25/2007  04:05 PM           185,938 victoria_veckham_shirt_sheer_5_big.jpg
10/19/2009  03:16 PM           120,415 victoria_veckham_shirt_sheer_5_big.jpg.paq8px
08/25/2007  03:47 PM           824,722 Wildwood_200615.JPG
10/19/2009  03:17 PM           642,257 Wildwood_200615.JPG.paq8px
08/25/2007  04:07 PM            18,167 wolk23.jpg
10/19/2009  03:17 PM            14,652 wolk23.jpg.paq8px
10/19/2009  02:44 PM            17,746 wolk23.jpg.zpaq1
10/19/2009  02:54 PM            17,277 wolk23.jpg.zpaq2
08/25/2007  03:51 PM           426,089 XXXA4I0020.JPG
10/19/2009  03:17 PM           266,366 XXXA4I0020.JPG.paq8px
             306 File(s)     61,918,306 bytes

[Jan Ondrus]

Fixed another bug in reading header (APP and COM field were not properly skipped).

[schnaader]

Fixed another bug in reading header (APP and COM field were not properly skipped).

I tried the fixed config files on the 47 files that failed before with ZPAQ v1.07. There are only 7 files left that still fail the preprocessing test, 2 of them (4215626041.jpg and EZEGNAX21.jpg) even gave a Windows "send bug report" dialog.

As a sidenote, a little ZPAQ usability request: It would be nice to have the size of the preprocessed file displayed, too (at the moment only the size of the original file is shown).

Results for jpg_test1.cfg:

20.10.2009  17:42           105.347 070730_Arcachon_22.jpg.zpaq1
20.10.2009  17:42            79.447 070806_Main_06.jpg.zpaq1
20.10.2009  17:45           532.444 110_1063.JPG.zpaq1
20.10.2009  17:46           143.711 AkhaSwingFest_SIMANN_10.JPG.zpaq1
20.10.2009  17:46            76.092 AkhaSwingFest_SIMANN_9.JPG.zpaq1
20.10.2009  17:47           284.295 august_18_2007_enso_karate_new_buffalo_beach_training_102.JPG.zpaq1
20.10.2009  17:49           284.107 buster_002.jpg.zpaq1
20.10.2009  17:50           105.291 catt_002.jpg.zpaq1
20.10.2009  17:51           111.987 chiangrai_068.jpg.zpaq1
20.10.2009  17:51           107.116 chiangrai_097.jpg.zpaq1
20.10.2009  17:53           266.075 CIMG4779.JPG.zpaq1
20.10.2009  17:53           186.254 Cold_Sunset.JPG.zpaq1
20.10.2009  17:54            38.000 co_022.jpg.zpaq1
20.10.2009  17:54            87.531 dad.jpg.zpaq1
20.10.2009  17:56           355.686 DSC02753.JPG.zpaq1
20.10.2009  17:57           248.303 DSC02837.JPG.zpaq1
20.10.2009  17:59           321.909 DSCN0400.JPG.zpaq1
20.10.2009  18:02           513.078 DSCN1937.JPG.zpaq1
20.10.2009  18:07           806.967 DSCN2865.JPG.zpaq1
20.10.2009  18:08           210.743 DSC_0020.JPG.zpaq1
20.10.2009  18:10           204.674 grey_002.jpg.zpaq1
20.10.2009  18:15           194.497 HPIM2867.JPG.zpaq1
20.10.2009  18:15            30.813 ILC_056.jpg.zpaq1
20.10.2009  18:16            36.062 ILC_072.jpg.zpaq1
20.10.2009  18:28         1.971.935 IMG_0010.JPG.zpaq1
20.10.2009  18:29           216.766 IMG_3969.JPG.zpaq1
20.10.2009  18:30           188.736 kohphangan4_014.jpg.zpaq1
20.10.2009  18:31            94.012 kohphangan_missingshots_029.jpg.zpaq1
20.10.2009  18:32           105.535 kohphangan_missingshots_033.jpg.zpaq1
20.10.2009  18:43         1.832.378 March_2007_014.jpg.zpaq1
20.10.2009  18:46           333.925 mountains_july_2007_022.JPG.zpaq1
20.10.2009  18:46            36.955 Mu_007.jpg.zpaq1
20.10.2009  18:47           219.911 Neridas_21st_015.jpg.zpaq1
20.10.2009  18:48           110.558 newyork2007_15.jpg.zpaq1
20.10.2009  18:48           137.271 Overworked.jpg.zpaq1
20.10.2009  18:50           161.425 PICT0947.JPG.zpaq1
20.10.2009  18:54           792.275 Reflection_GalleryPlayer.jpg.zpaq1
20.10.2009  18:54           106.280 shadz_017.jpg.zpaq1
20.10.2009  18:59           755.370 Wildwood_200615.JPG.zpaq1
20.10.2009  19:02           352.116 XXXA4I0020.JPG.zpaq1

Total size: 12.745.877 bytes

Results for jpg_test2.cfg:

20.10.2009  17:40           100.029 070730_Arcachon_22.jpg.zpaq2
20.10.2009  17:41            76.653 070806_Main_06.jpg.zpaq2
20.10.2009  16:00           498.477 110_1063.JPG.zpaq2
20.10.2009  16:03           136.365 AkhaSwingFest_SIMANN_10.JPG.zpaq2
20.10.2009  16:04            73.607 AkhaSwingFest_SIMANN_9.JPG.zpaq2
20.10.2009  16:05           268.815 august_18_2007_enso_karate_new_buffalo_beach_training_102.JPG.zpaq2
20.10.2009  16:07           273.048 buster_002.jpg.zpaq2
20.10.2009  16:09            98.768 catt_002.jpg.zpaq2
20.10.2009  16:09           108.730 chiangrai_068.jpg.zpaq2
20.10.2009  16:10           103.817 chiangrai_097.jpg.zpaq2
20.10.2009  16:12           250.471 CIMG4779.JPG.zpaq2
20.10.2009  16:13           175.720 Cold_Sunset.JPG.zpaq2
20.10.2009  16:14            35.616 co_022.jpg.zpaq2
20.10.2009  16:14            85.514 dad.jpg.zpaq2
20.10.2009  16:16           333.052 DSC02753.JPG.zpaq2
20.10.2009  16:18           237.455 DSC02837.JPG.zpaq2
20.10.2009  16:19           308.017 DSCN0400.JPG.zpaq2
20.10.2009  16:22           488.314 DSCN1937.JPG.zpaq2
20.10.2009  16:27           758.052 DSCN2865.JPG.zpaq2
20.10.2009  16:28           202.709 DSC_0020.JPG.zpaq2
20.10.2009  16:30           195.194 grey_002.jpg.zpaq2
20.10.2009  16:36           184.023 HPIM2867.JPG.zpaq2
20.10.2009  16:36            29.855 ILC_056.jpg.zpaq2
20.10.2009  16:36            34.496 ILC_072.jpg.zpaq2
20.10.2009  16:50         1.851.151 IMG_0010.JPG.zpaq2
20.10.2009  16:52           205.369 IMG_3969.JPG.zpaq2
20.10.2009  16:53           180.864 kohphangan4_014.jpg.zpaq2
20.10.2009  16:54            90.045 kohphangan_missingshots_029.jpg.zpaq2
20.10.2009  16:54           102.129 kohphangan_missingshots_033.jpg.zpaq2
20.10.2009  17:08         1.727.077 March_2007_014.jpg.zpaq2
20.10.2009  17:10           317.865 mountains_july_2007_022.JPG.zpaq2
20.10.2009  17:11            35.585 Mu_007.jpg.zpaq2
20.10.2009  17:12           210.072 Neridas_21st_015.jpg.zpaq2
20.10.2009  17:13           106.886 newyork2007_15.jpg.zpaq2
20.10.2009  17:16           142.404 Overworked.jpg.zpaq2
20.10.2009  17:19           156.408 PICT0947.JPG.zpaq2
20.10.2009  17:30           774.184 Reflection_GalleryPlayer.jpg.zpaq2
20.10.2009  17:30            99.059 shadz_017.jpg.zpaq2
20.10.2009  17:36           691.133 Wildwood_200615.JPG.zpaq2
20.10.2009  17:38           332.643 XXXA4I0020.JPG.zpaq2

Total size: 12.079.671 bytes

The 7 files that failed are:

4215626041.jpg
catnKeys200519.jpg
DSC_1825.jpg
EZEGNAX21.jpg
Hassam_GalleryPlayer.jpg
Image00006.jpg
victoria_veckham_shirt_sheer_5_big.jpg

[Jan Ondrus]

The 7 files that failed are:

4215626041.jpg
catnKeys200519.jpg
DSC_1825.jpg
EZEGNAX21.jpg
Hassam_GalleryPlayer.jpg
Image00006.jpg
victoria_veckham_shirt_sheer_5_big.jpg

1.)
- 4215626041.jpg
- EZEGNAX21.jpg
are JPEG progressive which is not supported. I fixed preprocessor so that it won't crash on this files but preprocessing will fail.
2.)
- catnKeys200519.jpg
- DSC_1825.jpg
- Hassam_GalleryPlayer.jpg
- Image00006.jpg
- victoria_veckham_shirt_sheer_5_big.jpg
failed due to bug : missing stuffed byte before RST code wasn't restored by ZPAQL code.

[Matt Mahoney]

Some test results on the Picasa set. For comparison purposes I added the uncompressed sizes of the 2 progressive mode JPEGs that jpg_test1/2 won't compress.

25,778,926 picasa original size of 100 files
25,559,350 stuffit 11 fast (18 sec)
25,415,338 zip -9 (3 sec)
23,114,270 zpaq ocjpg_test1.cfg (600 sec, fails on 4215626041.jpg, EZEGNAX21.jpg)
21,903,634 zpaq ocjpg_test2.cfg (840 sec, fails on 4215626041.jpg, EZEGNAX21.jpg)
20,951,499 packjpg 2.2 (62 sec)
20,627,622 paq8o -6 (916 sec)
20,438,131 paq8fthis3 -6 (738 sec)
20,292,713 paq8o6 -6 (1080 sec)
20,263,436 paq8o7 -6 (1089 sec)
20,244,413 paq8o8 -6 (1160 sec)
20,207,952 paq8px_v64 -6 (1140 sec)
19,819,783 stuffit 11 max (50 sec)
19,828,626 stuffit 12 --jpeg-no-thumbnails (58 sec)

In other news I'm working on a Linux/Mac port for zpaq 1.09. It mostly works as-is on a PowerPC except that Linux doesn't like "$" in the temporary file names so preprocessing fails. I'm also fixing a bunch of minor bugs mostly in the display and cleaning up compiler warnings.

[Matt Mahoney]

zpaq 1.09 is posted to http://mattmahoney.net/dc/

Mainly this is a Linux port and some mostly cosmetic bug fixes. The port involved changing the temporary file names (no $), flushing stdout when showing compression progress, and fixing a crash when neither $TEMP nor $TMPDIR is set. $TMPDIR is no longer used. All temporary files including preprocessing now go in %TEMP% in Windows or $TEMP in Linux. If you want them in /tmp then

setenv TEMP /tmp

or else they go in the current directory. %TEMP% is normally set in Windows to a temp directory.

Other changes:

q (quiet) option - no display during compression or decompression except for errors.

Compression progress shows preprocessed size if different, so you know when it will be finished.

Display is updated once per second regardless of speed.

Program no longer uses fseek() and ftell() anywhere, so file sizes should display correctly over 2 GB (not tested).

Optimized decompressers check the block header and don't assume it matches the compiled ZPAQL code.

Documentation in zpaq.cpp is updated to reflect these changes.

BTW, nice job on moving the forum to a reliable website.

[Matt Mahoney]

http://mattmahoney.net/dc/

I posted Jan Ondrus's JPEG model and added a few test results for his other models. The BMP model would be ranked #1 on maximumcompression.com. Too bad that site appears dead Also reorganized a bit and made zpaqsfx a separate download.

I also tested BWT (fast mode) with and without optimization on enwik9. Times below are with optimization (oc, ox). bwt_j2.cfg,18 without optimization (c, x) is 5593, 4347. Optimizing makes decompression 2.35 times faster.

  Config                          enwik8      enwik9       oc    ox   Mem
  ------                        ----------  -----------   ----  ----  ----
  mid.cfg                       20,941,558   180,279,221  3521  3652   111
  bwt_j2.cfg,18                 20,756,883   174,171,965  4529  1847  1838  (256 MB block)
  bwt_slowmodel_1GB_block.cfg   20,756,996   163,565,006  7957  3875  1443  (1 GB block)

More testing on zpaq 1.09 with large files (3 GB, 5 GB) shows everything works, but lzppre (preprocessor for min.cfg) fails above 4 GB. zpaq detects this and refuses to compress, just like it is supposed to. Haven't fixed yet.

[Matt Mahoney]

zpaq v1.10 is posted to http://mattmahoney.net/dc/

This is a minor bug fix for g++ 4.4.1. v1.09 failed during compression if a preprocessor was used (attempting to read from a closed input file). Otherwise there is no change in speed or compression or in any of the commands. The code in v1.09 should not have worked but apparently did anyway (in 3 different compilers). Another user found the bug when testing in Ubuntu and sent me a patch.

[mhajicek]

Hi,
thank you, nice archiver, or at least promising - especially considering the future archive compatibility. Do you plan to release a version that will be able to use more than 2GB RAM ? It seems current config files can be set so, but not the 32bit program. I think that having more than 2GB RAM is not / or at least will not be / so rare anymore, and so the archiver and users can gain smaller files easily sometimes.
Well, in general, I'm surprised how few archivers support more than 2GB RAM, good exception beeing 7-zip (and even 7-zip only for lzma algo, not for ppm), nanozip has a bug, ash is experimental, 64bit WinRK doesn't allow to go beyond 2GB model either, ... But I'm not a good/active programmer, so maybe I don't uderstand it just because I don't know that changing the archiver or algorythm in such a way is not easy at all. Is this the reason?
regards
Michal

[Matt Mahoney]

You should be able to compile ZPAQ for 64 bits and write models that use more than 2 GB memory although each component is limited to 1 GB in most cases. This is a limitation of using 32 bit arithmetic for memory allocation and computing array indexes.

Normally most memory would be used by the ICM and ISSE components. These map a context to a 1-byte bit history using hash tables indexed by by a 32 bit context hash computed every 4 bits of input. Each table entry is an array of 15 bit histories (for up to 3 bits of additional context) and an 8 bit checksum. The context hash is computed as a 32 bit number in the HCOMP section of the ZPAQL code. The low bits (up to 26 for 1 GB) index the table and the high 8 bits are used as a hash confirmation. This means for the largest size table there is a 2 bit overlap so there is an effective 6 bit checksum which means 1/64 chance of an undetected hash collision for each of the 3 entries in the hash search. Using larger tables would mean more false matches which would hurt compression.

A CM does not have this problem because it is indexed directly and hash collisions (always undetected) have a much smaller effect. It also has a 32 bit index and 4 bytes per entry so in theory it could be as large as 16 GB. However, using a large CM table for high order contexts usually gives worse compression than an ICM. A CM is better suited for low orders for stationary sources.

[willvarfar]

a crash when neither $TEMP nor $TMPDIR is set.

tmpfile() is the appropriate function and it works on both Windows and Linux.

[Matt Mahoney]

Actually tmpfile() fails in Windows Vista/g++ on the C: drive if you don't have administrator privileges. This is a problem in all recent PAQ versions and BBB in slow mode. tmpfile() tries to create a temporary file at the root of the current drive. I have to go to D: to run PAQ. tmpfile() works under XP. I haven't tried in Win7.

tmpnam() has the same problem, since it generates a file name starting with \

ZPAQ would be a lot simpler if I could use tmpfile() instead of writing my own code to work around a bug in g++. It is hard to make this thread safe (so I didn't).

[willvarfar]

crikey I didn't know that;

GetTempFileName() might be better, you can tell it to do it in the current directory

still, it shocks me that tmpfile() doesn't work on vista, but it seems you're all over it

[Matt Mahoney]

Yes, that would work but I would like to avoid OS dependency since it is Windows only. It would be better to not use temporary files at all. In zpaq I use it to check that running the input through the preprocessor and postprocessor restores the original file by comparing the SHA-1 checksum. This could be done in 2 passes without a temp file, or in 1 pass but you wouldn't find out until the file was already compressed, or could be skipped entirely if you trust that the code is correct. (You would find out when you decompressed and got a checksum error).