paq8px

[Jan Ondrus]

paq8px is paq version based on paq8p with improved/added models/detection from paq8p3

DIFFERENCES FROM PAQ8P

Added nestModel from paq8p3
Modified wordModel from paq8p3
Modified .pbm, .pgm, .ppm, .bmp, .rgb detection (from paq8p3)
Modified WAV model (from paq8p_)
Modified JPEG model (from paq8p2)
Renamed bmpModel (24-bit) to im24bitModel + .ppm compression added (from paq8p3)
Added bmpModel1 (1-bit) and PBM models (from paq8p3)
Removed pic model

http://www.sendspace.com/file/s23d54

[LovePimple]

Awesome! Thanks Jan!

Compiled...

EDIT: Attachment "paq8px_00.zip" removed.

[kaitz]

Well, . Someone had free time.

EDIT:

Some old problems are back.

G:\svn\PAQ8\trunk>paq8px -7 G:\svn\PAQ8\trunk\WcgopFsd.dll
Creating archive G:\svn\PAQ8\trunk\WcgopFsd.dll.paq8px with 1 file(s)...
G:/svn/PAQ8/trunk/WcgopFsd.dll 79385 -> Transform fails at 79385, input=0 decoded=0, skipping...
30247
79385 -> 30280
Time 13.97 sec, used 845993497 bytes of memory

G:\svn\PAQ8\trunk>paq8p3 -7 G:\svn\PAQ8\trunk\WcgopFsd.dll
Creating archive G:\svn\PAQ8\trunk\WcgopFsd.dll.paq8p3 with 1 file(s)...
G:/svn/PAQ8/trunk/WcgopFsd.dll 79385 -> 29774        (       29435)
79385 -> 29807
Time 8.44 sec, used 660464089 bytes of memory

Type    Size            Parts
unkn    23190           3
exe     56195           1
Total parts: 4, fragmentation cost 20 bytes

http://encode.su/forum/showthread.php?t=157

E2:
paq8p3 -7 world95.txt 2988578 -> 354308, Time 259.31 sec, used 803216697 bytes of memory
paq8px -7 world95.txt 2988578 -> 352722, Time 421.91 sec, used 845993497 bytes of memory

[Skymmer]

Jan, Kaitz (I suppose you're also Kaido), first of all I want to thank you for PAQs you have made. Good job!
Now, here are some tests I've conducted on speed optimized\sse p, p1, p3, px versions. Unfortunately I wasn't able to find speed optimized version of p1 so don't look too much into its time results. Tests have been performed on various types of files with both speed and resulting size measured. -6 level have been used. The value in brackets after a filename is the original size. Columns are PAQ version, time, size. After each test I gaved a little comment. Let's go...

EXE test

FreePascal.exe (5 392 896)
--------------------------
paq8p_sse2        640.188       1 228 793
paq8px_speed      742.881       1 226 706
paq8p1            483.468       1 248 517
paq8p3_speed      456.052       1 245 152

All clear here. px better and slower than p\p1\p3. p3 surprisingly not only faster but also better that p1. But I know one drawback of p3 for PE files and will show it in another thread.

BSP test

d1_trainstation_03.bsp (5 909 064)
----------------------------------
paq8p_sse2        602.452       1 277 647
paq8px_speed      706.456       1 272 868
paq8p1            505.017       1 283 404
paq8p3_speed      419.189       1 283 911

The test file is from the HL2 game. Same situation here except the fact that p3 now little bit larger.

TXT test

l200903.txt (3 964 732)
-----------------------
paq8p_sse2        423.205       83 146
paq8px_speed      503.735       82 363
paq8p1            349.203       89 304
paq8p3_speed      296.982       89 494

No comments.

JPEG test

DSC02315.JPG (4 828 816)
------------------------
paq8p_sse2        140.151       3 657 757
paq8px_speed      137.702       3 651 709
paq8p1            155.294       3 658 651
paq8p3_speed      134.589       3 651 767

Hmmm... In this test px not only faster than p but also best on size. p3 shows the best time.

WAV test

Squarepusher - My Red Hot Car.wav (4 769 440) (27 sec. loop)
------------------------------------------------------------
paq8p_sse2        *******       *********
paq8p             121.126       2 721 473
paq8px_speed      185.152       2 709 594
paq8p1            165.332       2 721 467
paq8p3_speed      133.727       2 721 466

This test brings up some unpleasant and strange things. First of all sse2 of p just crashes on .wav files so I've done a test with plain compile.
px is best on size. p3 is fastest and 1 byte smaller than p1.

BMP test

Back.bmp (3 120 656)
--------------------
paq8p_sse2        53.646        315 486
paq8px_speed      48.408        275 871
paq8p1            50.376        320 204
paq8p3_speed      51.266        313 206

Nice! px not only the best on size but also the fastest ! Just look at the size gap.

The good thing I love in tests is the possibility to find out some interesting issues. The BMP test made it for me. I'm not talking about good results of px, just listen...
I'm always thought that BMP format have fixed-lenght header but seems I was wrong. Looks like "ever live - ever learn" situation. When I was searching for test file I found 4 different file sizes for files with same 24 bit depth and same 1024x768 resolution. It was 2 359 296, 2 359 350, 2 359 352 and 2 359 362 bytes. The only clue I have is that only one file (2 359 352) showed up 72x72 DPI value while others showed unknown value in XnView.

The worst thing about it, is that not all of the files have been correctly detected and accepted. I suppose PAQ8 BMP detection is little bit unoptimal. So let's test. Yes\No means if file detected as BMP or not.

            | paq8p_sse2 | paq8px_speed | paq8p1 | paq8p3_speed |
----------------------------------------------------------------
2 359 350   |      yes   |     yes      |   yes  |      yes     |
----------------------------------------------------------------
2 359 352   |      yes   |     yes      |   yes  |      yes     |
----------------------------------------------------------------
2 359 362   |      no    |     yes      |   no   |      no      |
----------------------------------------------------------------

EDIT 26 Apr 2009 21:20 :
- Updated JPEG results for px
- Modified BMP parsing table. Removed 2 359 296 file since its just corrupted one.

OBSOLETE:
Furthermore, for p and px 2 359 296 file gaved the following message:

Transform fails at 2359296, input=11 decoded=11, skipping...

Well. that is all

[kaitz]

Main diff between px and p3 version: px uses all models for data (except images, wav) so it will have most time better results on binary and text data.
p3 version text detection will use min 80 byte length text. So if binary and mixed text (smaller than 80 bytes) in effect then p3 will be worse. I wanted to give speed to p3 and other versions of same logic.
Wav,jpg,bmp24 models are different on px and p3, that's the size diff.

I thought that the best option would be completely separated text from the mixed data. In a separate file, which contains the indices for the original file. This is the one with all the text. And other is binary. Increasing the size of course.

I think we do not need to process text with all possible models. Only those that are made for textual data.

I no it is not optimal for compression. Like when XORng text data p3 will perform bad.

I have not tested but i think jpg model and exe model (a la) in one mixer is not good either. Make separate mixers and create threaded app where each thread compresses data that is for him. Like for each data there is small paq like compressor(thread) (Idea only).
Or i am just crayzy.

[Jan Ondrus]

I couldn't find such 2 359 296 bytes BMP file. Is it correctly displayed in image viewer? 2359296 is size of raw data for 1024x768 image (1024*768*3=2359296)

[Jan Ondrus]

fixed
-WcgopFsd.dll
-2359362 bytes 1024x768 bitmap
-JPEG model wasn't replaced in previous version!

http://www.sendspace.com/file/qtc6o3

[Skymmer]

I couldn't find such 2 359 296 bytes BMP file. Is it correctly displayed in image viewer? 2359296 is size of raw data for 1024x768 image (1024*768*3=2359296)

Hmmm. Nice observation. Actualy this file is screenshot maded by BioWare's Mass Effect game. It looks normally in XnView\IrfanView\MSPaint but refused by internal XP's Image and Fax viewer. Looking in HEX into this file shows up that there is a header so I believe that some last bytes just missing. Well, seems we found a bug in Mass Effect
By the way, is it possible to create a parser which will work on such partially corrupted files ? I mean PAQ will "filter" data until bad pattern is found and then just copy remaining data or pack it using another model.

[LovePimple]

fixed
-WcgopFsd.dll
-2359362 bytes 1024x768 bitmap
-JPEG model wasn't replaced in previous version!

http://www.sendspace.com/file/qtc6o3

Thanks Jan!

Compiled...

EDIT: Attachment "paq8px_01.zip" removed.

[Skymmer]

I've updated the test table. LovePimple, thanks for compiles.

[Jan Ondrus]

SFC compressed using -6 switch:

              paq8p -6  paq8p3 -6 paq8px -6
vcfiu.hlp     402041    411485    400352
world95.txt   360459    356602    355169
mso97.dll     1301443   1308919   1300814
A10.jpg       638202    637199    637194
acrord32.exe  915251    924571    913731
rafale.bmp    539077    546486    539098
english.dic   385991    389881    386264
fp.log        266534    260537    258423
ohs.doc       483784    486201    483349
FlashMX.pdf   3559759   3561194   3557052
total         8852541   8883075   8831446
%             100,00%   +0,34%    -0,24%

[Darek]

Hi!
Again big thanks for this version of paq8px. For my testbed it's actually the best version for overall test. Really great job!
For comparison attach sample of my testscores for selected - previous and best versions of paq.

file paq8p | paq8p1 | paq8p1_
0.WAV 1341247 | 1341330 | 1333319
1.BMP 264191 | 266187 | 266187
A.TIF 869702 | 879181 | 879181
B.TGA 818050 | 827371 | 827371
C.TIF 335089 | 335918 | 335918
D.TGA 310368 | 310716 | 310716
E.TIF 499318 | 499308 | 499308
F.JPG 90098 | 90098 | 90098
G.EXE 1366723 | 1368939 | 1368939
H.EXE 484633 | 489860 | 489860
I.EXE 216357 | 217258 | 217258
J.EXE 43147 | 43138 | 43138
K.WAD 2619296 | 2633806 | 2633806
L.PAK 2715542 | 2721579 | 2704501
M.DBF 51897 | 64108 | 64108
N.ADX 86029 | 86628 | 86628
O.APR 3868 | 3889 | 3889
P.FM3 1026 | 1020 | 1020
Q.WK3 206910 | 211116 | 211116
R.DOC 28614 | 28505 | 28505
S.DOC 25224 | 25008 | 25008
T.DOC 18085 | 17914 | 17914
U.DOC 8426 | 8371 | 8371
V.DOC 17923 | 17825 | 17825
W.DOC 12859 | 12822 | 12822
X.DOC 10782 | 10728 | 10728
Y.CFG 322 | 320 | 320
Z.MSG 178 | 177 | 177
total 12445904 | 12513120 | 12488031

file paq8p2 | paq8p3 | paq8px
0.WAV 1342234 | 1340445 | 1332631
1.BMP 266187 | 265835 | 264192
A.TIF 896031 | 886909 | 864257
B.TGA 845515 | 839142 | 813144
C.TIF 336121 | 336048 | 334985
D.TGA 313445 | 310461 | 309715
E.TIF 499371 | 499252 | 499032
F.JPG 89750 | 89750 | 89751
G.EXE 1371761 | 1368934 | 1364363
H.EXE 500004 | 488442 | 481886
I.EXE 218726 | 216896 | 215846
J.EXE 43148 | 43141 | 43064
K.WAD 2650698 | 2644013 | 2609474
L.PAK 2723700 | 2725091 | 2698082
M.DBF 64183 | 63966 | 51430
N.ADX 87646 | 86828 | 85546
O.APR 3890 | 3897 | 3851
P.FM3 1023 | 1559 | 1027
Q.WK3 221922 | 211588 | 204791
R.DOC 28506 | 28437 | 28409
S.DOC 25012 | 25056 | 24951
T.DOC 17920 | 17869 | 17823
U.DOC 8374 | 8362 | 8357
V.DOC 17828 | 17788 | 17760
W.DOC 12826 | 12801 | 12787
X.DOC 10731 | 10728 | 10719
Y.CFG 321 | 321 | 321
Z.MSG 177 | 174 | 178
total 12597050 | 12543733 | 12388372

file best score | best version
0.WAV 1332631 | paq8px
1.BMP 264191 | paq8p
A.TIF 847825 | paq8k
B.TGA 797003 | paq8k
C.TIF 334741 | paq8k
D.TGA 308679 | paq8k
E.TIF 498721 | paq8k
F.JPG 89750 | paq8p2
G.EXE 1363231 | paq8k
H.EXE 470414 | paq8k
I.EXE 212170 | paq8k
J.EXE 42963 | paq8k
K.WAD 2589976 | paq8k
L.PAK 2698082 | paq8px
M.DBF 51430 | paq8px
N.ADX 84850 | paq8k
O.APR 3851 | paq8px
P.FM3 1010 | paq8k
Q.WK3 192158 | paq8k
R.DOC 25612 | paq8hp12
S.DOC 24217 | paq8hp12
T.DOC 16547 | paq8hp12
U.DOC 7372 | paq8hp12
V.DOC 16260 | paq8hp12
W.DOC 11191 | paq8hp12
X.DOC 9574 | paq8hp12
Y.CFG 283 | paq8hp12
Z.MSG 109 | paq8hp12
total 12388372 | paq8px

Beste regards
Darek

[Darek]

Again
comparison of original file, paq8k and paqhp12 - it dissapear from my last post.

file original | paq8k | paq8hp12
0.WAV 2762044 | 1410378 | 1441483
1.BMP 600054 | 298499 | 336511
A.TIF 2870552 | 847825 | 913637
B.TGA 2872818 | 797003 | 863576
C.TIF 896094 | 334741 | 349948
D.TGA 721185 | 308679 | 333179
E.TIF 506390 | 498721 | 503029
F.JPG 112038 | 95318 | 112427
G.EXE 2069376 | 1363231 | 1376991
H.EXE 1828544 | 470414 | 503187
I.EXE 715493 | 212170 | 238759
J.EXE 46665 | 42963 | 43347
K.WAD 12408292 | 2589976 | 2881672
L.PAK 7764079 | 2809607 | 3043524
M.DBF 1918940 | 53325 | 69673
N.ADX 531456 | 84850 | 91462
O.APR 32256 | 3882 | 4064
P.FM3 183841 | 1010 | 1367
Q.WK3 994350 | 192158 | 267717
R.DOC 120201 | 28685 | 25612
S.DOC 141417 | 25327 | 24217
T.DOC 76898 | 18096 | 16547
U.DOC 34572 | 8496 | 7372
V.DOC 88541 | 18013 | 16260
W.DOC 49544 | 12948 | 11191
X.DOC 36977 | 10882 | 9574
Y.CFG 800 | 349 | 283
Z.MSG 289 | 214 | 109
total 40383706 | 12537760 | 13486718

Darek

[kaitz]

Darek, can you provide timing tables.

[Darek]

Kaitz,
time is absolutely not important factor of my tests and I didn't store it in my scoretables. From other hand some tests was made on different machines.
But there isn't any problem to retest it again and find these times. I'll do it.

Darek

[Stephan Busch]

paq8px_speed_optimised.exe from 26th of April crashes on my bitmap set
at about 50 MB (vista message: "paq8px_speed_optimised.exe has encountered a problem and needs to be closed")

No problems on the other sets tested so far.


Regards,
Stephan

[Jan Ondrus]

fixed bug in pgm/pbm/ppm detection

http://www.sendspace.com/file/ubbmyo

edited:
there was one minor problem in previous version
m8.pgm 12325 -> PGM 128x966911
should be
m8.pgm 12325 -> PGM 128x96 6911

[LovePimple]

Thanks Jan!

Compiled...

EDIT: Attachment "paq8px_02.zip" removed.

[Stephan Busch]

Thanks for bugfixing, Jan. This one seems to run without errors.

Thanks for compiling, LovePimple.

[Darek]

Kaitz,
please find attached file with test scores with timings. Paq8p3 (speed optimized) is obviously the best - about 2 times faster than Paq8px.

Surprisingly very good times got Paq8hp12, especially when we realize that it convert standard files into text and sizes of these converted files are bigger (for non-textual files). Additionaly if we would adjust time for building dictionary (about 45-50 sek.) and time for converting, then for most of files, compression speed will be similar or higher than Paq8p3!

Of course, my favourite paq8k was the worst in time comparison - about 3 times slower than paq8px, but it isn't a major indicator for me.

Darek

[pat357]

Kaitz,
please find attached file with test scores with timings. Paq8p3 (speed optimized) is obviously the best - about 2 times faster than Paq8px.

Of course, my favourite paq8k was the worst in time comparison - about 3 times slower than paq8px, but it isn't a major indicator for me.

Thanks for your testings, Darek !
BTW, I can't find a link to download PAQ8K anymore...
Could you post it here, please ?
Thanks !

[Mike]

Thanks for your testings, Darek !
BTW, I can't find a link to download PAQ8K anymore...
Could you post it here, please ?
Thanks !

You can find it on Matt's PAQ page.

Direct links :
http://ilovemyking.googlepages.com/paq8k.zip
http://ilovemyking.googlepages.com/paq8k2.zip
http://ilovemyking.googlepages.com/paq8k2_src.zip

[Darek]

Hi!
here you are paq8k.

Paq8k2 is a special edition of Paq8k, made for highly non-redundant files. It set a new record in Matt Mahoney Generic Compression Benchmark.
Unfortunatelly for other files from my testbed average compression is about 11% worse than Paq8k. And it is dramaticly slow.

Darek

[Stephan Busch]

the provided PAQ8PX now freezes at 535 MB of the PPM/PGM testset.
the bit counter does not go on for over 5 hours but the task manager tells me that PAQ8PX still keeps the cpu busy. the same problem occured on the cd image testset.

[kaitz]

Kaitz,
please find attached file with test scores with timings. Paq8p3 (speed optimized) is obviously the best - about 2 times faster than Paq8px.

Surprisingly very good times got Paq8hp12, especially when we realize that it convert standard files into text and sizes of these converted files are bigger (for non-textual files). Additionaly if we would adjust time for building dictionary (about 45-50 sek.) and time for converting, then for most of files, compression speed will be similar or higher than Paq8p3!

Of course, my favourite paq8k was the worst in time comparison - about 3 times slower than paq8px, but it isn't a major indicator for me.

Darek

Nice.

[Jan Ondrus]

There was problem with compressing files larger than 1 block (512MB for -7 switch, 4MB for -0 swtich).
I fixed that problem and increased size of block 2x.
Please test if bug is properly fixed.

[LovePimple]

Thanks Jan!

Compiled...

EDIT: Attachment "paq8px_v4.zip" removed.

[LovePimple]

Paq8px is the new compression King...

http://www.maximumcompression.com/

[Jan Ondrus]

paq8px_v5
- small modifications in exe detection and e8/e9 exe filter

               paq8px_v4 -6   paq8px_v5 -6
mso97.dll      1300814        1294239 (-0.5%)
acrord32.exe   913731         908313 (-0.6%)
mptrack.exe    346576         346119 (-0.1%)
i_view32.exe   298651         298329 (-0.1%)

[LovePimple]

Awesome! Thanks Jan!

Compiled...

EDIT: Attachment "paq8px_v5.zip" removed.