Page 1 of 44 12311 ... LastLast
Results 1 to 30 of 1938

Thread: paq8px

Hybrid View

Previous Post Previous Post   Next Post Next Post
  1. #1
    Programmer Jan Ondrus's Avatar
    Join Date
    Sep 2008
    Location
    Rychnov nad Kněžnou, Czech Republic
    Posts
    279
    Thanks
    33
    Thanked 138 Times in 50 Posts

    paq8px

    paq8px is paq version based on paq8p with improved/added models/detection from paq8p3

    DIFFERENCES FROM PAQ8P

    Added nestModel from paq8p3
    Modified wordModel from paq8p3
    Modified .pbm, .pgm, .ppm, .bmp, .rgb detection (from paq8p3)
    Modified WAV model (from paq8p_)
    Modified JPEG model (from paq8p2)
    Renamed bmpModel (24-bit) to im24bitModel + .ppm compression added (from paq8p3)
    Added bmpModel1 (1-bit) and PBM models (from paq8p3)
    Removed pic model

    http://www.sendspace.com/file/s23d54

  2. #2
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts

    Thumbs up

    Awesome! Thanks Jan!

    Compiled...

    EDIT: Attachment "paq8px_00.zip" removed.

  3. #3
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    509
    Thanks
    208
    Thanked 347 Times in 185 Posts

    Thumbs up

    Well, . Someone had free time.

    EDIT:

    Some old problems are back.
    Code:
    G:\svn\PAQ8\trunk>paq8px -7 G:\svn\PAQ8\trunk\WcgopFsd.dll
    Creating archive G:\svn\PAQ8\trunk\WcgopFsd.dll.paq8px with 1 file(s)...
    G:/svn/PAQ8/trunk/WcgopFsd.dll 79385 -> Transform fails at 79385, input=0 decoded=0, skipping...
    30247
    79385 -> 30280
    Time 13.97 sec, used 845993497 bytes of memory
    
    G:\svn\PAQ8\trunk>paq8p3 -7 G:\svn\PAQ8\trunk\WcgopFsd.dll
    Creating archive G:\svn\PAQ8\trunk\WcgopFsd.dll.paq8p3 with 1 file(s)...
    G:/svn/PAQ8/trunk/WcgopFsd.dll 79385 -> 29774        (       29435)
    79385 -> 29807
    Time 8.44 sec, used 660464089 bytes of memory
    
    Type    Size            Parts
    unkn    23190           3
    exe     56195           1
    Total parts: 4, fragmentation cost 20 bytes
    http://encode.su/forum/showthread.php?t=157

    E2:
    paq8p3 -7 world95.txt 2988578 -> 354308, Time 259.31 sec, used 803216697 bytes of memory
    paq8px -7 world95.txt 2988578 -> 352722, Time 421.91 sec, used 845993497 bytes of memory
    Last edited by kaitz; 25th April 2009 at 23:26.
    KZo


  4. #4
    Member Skymmer's Avatar
    Join Date
    Mar 2009
    Location
    Russia
    Posts
    688
    Thanks
    41
    Thanked 173 Times in 88 Posts
    Jan, Kaitz (I suppose you're also Kaido), first of all I want to thank you for PAQs you have made. Good job!
    Now, here are some tests I've conducted on speed optimized\sse p, p1, p3, px versions. Unfortunately I wasn't able to find speed optimized version of p1 so don't look too much into its time results. Tests have been performed on various types of files with both speed and resulting size measured. -6 level have been used. The value in brackets after a filename is the original size. Columns are PAQ version, time, size. After each test I gaved a little comment. Let's go...

    EXE test

    Code:
    FreePascal.exe (5 392 896)
    --------------------------
    paq8p_sse2        640.188       1 228 793
    paq8px_speed      742.881       1 226 706
    paq8p1            483.468       1 248 517
    paq8p3_speed      456.052       1 245 152
    All clear here. px better and slower than p\p1\p3. p3 surprisingly not only faster but also better that p1. But I know one drawback of p3 for PE files and will show it in another thread.

    BSP test

    Code:
    d1_trainstation_03.bsp (5 909 064)
    ----------------------------------
    paq8p_sse2        602.452       1 277 647
    paq8px_speed      706.456       1 272 868
    paq8p1            505.017       1 283 404
    paq8p3_speed      419.189       1 283 911
    The test file is from the HL2 game. Same situation here except the fact that p3 now little bit larger.

    TXT test

    Code:
    l200903.txt (3 964 732)
    -----------------------
    paq8p_sse2        423.205       83 146
    paq8px_speed      503.735       82 363
    paq8p1            349.203       89 304
    paq8p3_speed      296.982       89 494
    No comments.

    JPEG test

    Code:
    DSC02315.JPG (4 828 816)
    ------------------------
    paq8p_sse2        140.151       3 657 757
    paq8px_speed      137.702       3 651 709
    paq8p1            155.294       3 658 651
    paq8p3_speed      134.589       3 651 767
    Hmmm... In this test px not only faster than p but also best on size. p3 shows the best time.

    WAV test

    Code:
    Squarepusher - My Red Hot Car.wav (4 769 440) (27 sec. loop)
    ------------------------------------------------------------
    paq8p_sse2        *******       *********
    paq8p             121.126       2 721 473
    paq8px_speed      185.152       2 709 594
    paq8p1            165.332       2 721 467
    paq8p3_speed      133.727       2 721 466
    This test brings up some unpleasant and strange things. First of all sse2 of p just crashes on .wav files so I've done a test with plain compile.
    px is best on size. p3 is fastest and 1 byte smaller than p1.

    BMP test

    Code:
    Back.bmp (3 120 656)
    --------------------
    paq8p_sse2        53.646        315 486
    paq8px_speed      48.408        275 871
    paq8p1            50.376        320 204
    paq8p3_speed      51.266        313 206
    Nice! px not only the best on size but also the fastest ! Just look at the size gap.

    The good thing I love in tests is the possibility to find out some interesting issues. The BMP test made it for me. I'm not talking about good results of px, just listen...
    I'm always thought that BMP format have fixed-lenght header but seems I was wrong. Looks like "ever live - ever learn" situation. When I was searching for test file I found 4 different file sizes for files with same 24 bit depth and same 1024x768 resolution. It was 2 359 296, 2 359 350, 2 359 352 and 2 359 362 bytes. The only clue I have is that only one file (2 359 352) showed up 72x72 DPI value while others showed unknown value in XnView.

    The worst thing about it, is that not all of the files have been correctly detected and accepted. I suppose PAQ8 BMP detection is little bit unoptimal. So let's test. Yes\No means if file detected as BMP or not.

    Code:
                | paq8p_sse2 | paq8px_speed | paq8p1 | paq8p3_speed |
    ----------------------------------------------------------------
    2 359 350   |      yes   |     yes      |   yes  |      yes     |
    ----------------------------------------------------------------
    2 359 352   |      yes   |     yes      |   yes  |      yes     |
    ----------------------------------------------------------------
    2 359 362   |      no    |     yes      |   no   |      no      |
    ----------------------------------------------------------------
    EDIT 26 Apr 2009 21:20 :
    - Updated JPEG results for px
    - Modified BMP parsing table. Removed 2 359 296 file since its just corrupted one.

    OBSOLETE:
    Furthermore, for p and px 2 359 296 file gaved the following message:
    Code:
    Transform fails at 2359296, input=11 decoded=11, skipping...
    Well. that is all
    Last edited by Skymmer; 26th April 2009 at 20:21.

  5. #5
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    509
    Thanks
    208
    Thanked 347 Times in 185 Posts

    Post

    Main diff between px and p3 version: px uses all models for data (except images, wav) so it will have most time better results on binary and text data.
    p3 version text detection will use min 80 byte length text. So if binary and mixed text (smaller than 80 bytes) in effect then p3 will be worse. I wanted to give speed to p3 and other versions of same logic.
    Wav,jpg,bmp24 models are different on px and p3, that's the size diff.

    I thought that the best option would be completely separated text from the mixed data. In a separate file, which contains the indices for the original file. This is the one with all the text. And other is binary. Increasing the size of course.

    I think we do not need to process text with all possible models. Only those that are made for textual data.

    I no it is not optimal for compression. Like when XORng text data p3 will perform bad.

    I have not tested but i think jpg model and exe model (a la) in one mixer is not good either. Make separate mixers and create threaded app where each thread compresses data that is for him. Like for each data there is small paq like compressor(thread) (Idea only).
    Or i am just crayzy.
    KZo


  6. #6
    Programmer Jan Ondrus's Avatar
    Join Date
    Sep 2008
    Location
    Rychnov nad Kněžnou, Czech Republic
    Posts
    279
    Thanks
    33
    Thanked 138 Times in 50 Posts
    I couldn't find such 2 359 296 bytes BMP file. Is it correctly displayed in image viewer? 2359296 is size of raw data for 1024x768 image (1024*768*3=2359296)

  7. #7
    Programmer Jan Ondrus's Avatar
    Join Date
    Sep 2008
    Location
    Rychnov nad Kněžnou, Czech Republic
    Posts
    279
    Thanks
    33
    Thanked 138 Times in 50 Posts
    SFC compressed using -6 switch:
    Code:
                  paq8p -6  paq8p3 -6 paq8px -6
    vcfiu.hlp     402041    411485    400352
    world95.txt   360459    356602    355169
    mso97.dll     1301443   1308919   1300814
    A10.jpg       638202    637199    637194
    acrord32.exe  915251    924571    913731
    rafale.bmp    539077    546486    539098
    english.dic   385991    389881    386264
    fp.log        266534    260537    258423
    ohs.doc       483784    486201    483349
    FlashMX.pdf   3559759   3561194   3557052
    total         8852541   8883075   8831446
    %             100,00%   +0,34%    -0,24%

  8. #8
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,149
    Thanks
    703
    Thanked 455 Times in 352 Posts

    paq8px comparison

    Hi!
    Again big thanks for this version of paq8px. For my testbed it's actually the best version for overall test. Really great job!
    For comparison attach sample of my testscores for selected - previous and best versions of paq.

    file paq8p | paq8p1 | paq8p1_
    0.WAV 1341247 | 1341330 | 1333319
    1.BMP 264191 | 266187 | 266187
    A.TIF 869702 | 879181 | 879181
    B.TGA 818050 | 827371 | 827371
    C.TIF 335089 | 335918 | 335918
    D.TGA 310368 | 310716 | 310716
    E.TIF 499318 | 499308 | 499308
    F.JPG 90098 | 90098 | 90098
    G.EXE 1366723 | 1368939 | 1368939
    H.EXE 484633 | 489860 | 489860
    I.EXE 216357 | 217258 | 217258
    J.EXE 43147 | 43138 | 43138
    K.WAD 2619296 | 2633806 | 2633806
    L.PAK 2715542 | 2721579 | 2704501
    M.DBF 51897 | 64108 | 64108
    N.ADX 86029 | 86628 | 86628
    O.APR 3868 | 3889 | 3889
    P.FM3 1026 | 1020 | 1020
    Q.WK3 206910 | 211116 | 211116
    R.DOC 28614 | 28505 | 28505
    S.DOC 25224 | 25008 | 25008
    T.DOC 18085 | 17914 | 17914
    U.DOC 8426 | 8371 | 8371
    V.DOC 17923 | 17825 | 17825
    W.DOC 12859 | 12822 | 12822
    X.DOC 10782 | 10728 | 10728
    Y.CFG 322 | 320 | 320
    Z.MSG 178 | 177 | 177
    total 12445904 | 12513120 | 12488031

    file paq8p2 | paq8p3 | paq8px
    0.WAV 1342234 | 1340445 | 1332631
    1.BMP 266187 | 265835 | 264192
    A.TIF 896031 | 886909 | 864257
    B.TGA 845515 | 839142 | 813144
    C.TIF 336121 | 336048 | 334985
    D.TGA 313445 | 310461 | 309715
    E.TIF 499371 | 499252 | 499032
    F.JPG 89750 | 89750 | 89751
    G.EXE 1371761 | 1368934 | 1364363
    H.EXE 500004 | 488442 | 481886
    I.EXE 218726 | 216896 | 215846
    J.EXE 43148 | 43141 | 43064
    K.WAD 2650698 | 2644013 | 2609474
    L.PAK 2723700 | 2725091 | 2698082
    M.DBF 64183 | 63966 | 51430
    N.ADX 87646 | 86828 | 85546
    O.APR 3890 | 3897 | 3851
    P.FM3 1023 | 1559 | 1027
    Q.WK3 221922 | 211588 | 204791
    R.DOC 28506 | 28437 | 28409
    S.DOC 25012 | 25056 | 24951
    T.DOC 17920 | 17869 | 17823
    U.DOC 8374 | 8362 | 8357
    V.DOC 17828 | 17788 | 17760
    W.DOC 12826 | 12801 | 12787
    X.DOC 10731 | 10728 | 10719
    Y.CFG 321 | 321 | 321
    Z.MSG 177 | 174 | 178
    total 12597050 | 12543733 | 12388372

    file best score | best version
    0.WAV 1332631 | paq8px
    1.BMP 264191 | paq8p
    A.TIF 847825 | paq8k
    B.TGA 797003 | paq8k
    C.TIF 334741 | paq8k
    D.TGA 308679 | paq8k
    E.TIF 498721 | paq8k
    F.JPG 89750 | paq8p2
    G.EXE 1363231 | paq8k
    H.EXE 470414 | paq8k
    I.EXE 212170 | paq8k
    J.EXE 42963 | paq8k
    K.WAD 2589976 | paq8k
    L.PAK 2698082 | paq8px
    M.DBF 51430 | paq8px
    N.ADX 84850 | paq8k
    O.APR 3851 | paq8px
    P.FM3 1010 | paq8k
    Q.WK3 192158 | paq8k
    R.DOC 25612 | paq8hp12
    S.DOC 24217 | paq8hp12
    T.DOC 16547 | paq8hp12
    U.DOC 7372 | paq8hp12
    V.DOC 16260 | paq8hp12
    W.DOC 11191 | paq8hp12
    X.DOC 9574 | paq8hp12
    Y.CFG 283 | paq8hp12
    Z.MSG 109 | paq8hp12
    total 12388372 | paq8px

    Beste regards
    Darek

  9. #9
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,149
    Thanks
    703
    Thanked 455 Times in 352 Posts
    Again
    comparison of original file, paq8k and paqhp12 - it dissapear from my last post.

    file original | paq8k | paq8hp12
    0.WAV 2762044 | 1410378 | 1441483
    1.BMP 600054 | 298499 | 336511
    A.TIF 2870552 | 847825 | 913637
    B.TGA 2872818 | 797003 | 863576
    C.TIF 896094 | 334741 | 349948
    D.TGA 721185 | 308679 | 333179
    E.TIF 506390 | 498721 | 503029
    F.JPG 112038 | 95318 | 112427
    G.EXE 2069376 | 1363231 | 1376991
    H.EXE 1828544 | 470414 | 503187
    I.EXE 715493 | 212170 | 238759
    J.EXE 46665 | 42963 | 43347
    K.WAD 12408292 | 2589976 | 2881672
    L.PAK 7764079 | 2809607 | 3043524
    M.DBF 1918940 | 53325 | 69673
    N.ADX 531456 | 84850 | 91462
    O.APR 32256 | 3882 | 4064
    P.FM3 183841 | 1010 | 1367
    Q.WK3 994350 | 192158 | 267717
    R.DOC 120201 | 28685 | 25612
    S.DOC 141417 | 25327 | 24217
    T.DOC 76898 | 18096 | 16547
    U.DOC 34572 | 8496 | 7372
    V.DOC 88541 | 18013 | 16260
    W.DOC 49544 | 12948 | 11191
    X.DOC 36977 | 10882 | 9574
    Y.CFG 800 | 349 | 283
    Z.MSG 289 | 214 | 109
    total 40383706 | 12537760 | 13486718

    Darek

  10. #10
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    509
    Thanks
    208
    Thanked 347 Times in 185 Posts

    Question

    Darek, can you provide timing tables.
    KZo


  11. #11
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,149
    Thanks
    703
    Thanked 455 Times in 352 Posts
    Kaitz,
    time is absolutely not important factor of my tests and I didn't store it in my scoretables. From other hand some tests was made on different machines.
    But there isn't any problem to retest it again and find these times. I'll do it.

    Darek

  12. #12
    Programmer Jan Ondrus's Avatar
    Join Date
    Sep 2008
    Location
    Rychnov nad Kněžnou, Czech Republic
    Posts
    279
    Thanks
    33
    Thanked 138 Times in 50 Posts
    paq8px_v5
    - small modifications in exe detection and e8/e9 exe filter

    Code:
                   paq8px_v4 -6   paq8px_v5 -6
    mso97.dll      1300814        1294239 (-0.5%)
    acrord32.exe   913731         908313 (-0.6%)
    mptrack.exe    346576         346119 (-0.1%)
    i_view32.exe   298651         298329 (-0.1%)
    Last edited by Jan Ondrus; 11th June 2009 at 13:04.

  13. #13
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts

    Thumbs up

    Awesome! Thanks Jan!

    Compiled...

    EDIT: Attachment "paq8px_v5.zip" removed.

  14. #14
    Member Skymmer's Avatar
    Join Date
    Mar 2009
    Location
    Russia
    Posts
    688
    Thanks
    41
    Thanked 173 Times in 88 Posts
    LovePimple, sorry for asking, but could you make PAQ8px_v5 compilation optimized for executable size, if it is not too difficult ?

  15. Thanks:

    Gonzalo (25th February 2018)

  16. #15
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Just a compilation, do further optimizations yourself.
    Attached Files Attached Files

  17. Thanks:

    Gonzalo (25th February 2018)

  18. #16
    Member Skymmer's Avatar
    Join Date
    Mar 2009
    Location
    Russia
    Posts
    688
    Thanks
    41
    Thanked 173 Times in 88 Posts
    Quote Originally Posted by m^2 View Post
    Just a compilation, do further optimizations yourself.
    Damn, you read my thoughts
    Big thanx for compilation

  19. Thanks:

    Gonzalo (25th February 2018)

  20. #17
    Tester
    Stephan Busch's Avatar
    Join Date
    May 2008
    Location
    Bremen, Germany
    Posts
    876
    Thanks
    474
    Thanked 175 Times in 85 Posts
    why has the paq8px_v5.zip attachment been removed?

    This version runs over my bitmap and cd image sets like a charm
    and seems to enter the top rank on Squeeze Chart.

    So far, the cross-platform binary set compresses as follows:

    paq8px_v4 -7 = 20.629.710 bytes
    paq8px_v5 -7 = 20.571.295 bytes

    very well done, guys.

  21. #18
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Quote Originally Posted by Stephan Busch View Post
    why has the paq8px_v5.zip attachment been removed?
    Because paq8px_v6 is exactly the same apart from a minor bug fix.

    Quote Originally Posted by Jan Ondrus View Post
    paq8px_v6
    -fixed: compressing using -8 switch was broken
    -fixed: you could use (not working) -9 switch in previous versions

  22. #19
    Member
    Join Date
    Oct 2007
    Location
    Germany, Hamburg
    Posts
    409
    Thanks
    0
    Thanked 5 Times in 5 Posts
    Same fix as done for paqp3.
    Bmp detection for negative height was broken

    See paq8p3 posting for details.

    I miss the nice statistics from paq8p3 in paq8px.
    Code:
    Type    Size            Parts
    unkn    480566          1
    bmp8    512             1
    Total parts: 2, fragmentation cost 10 bytes
    Attached Files Attached Files

  23. #20
    Member
    Join Date
    Nov 2010
    Location
    Earth
    Posts
    1
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Hmm, i cannot compile this on my own, using codeblocks and g++ v4.5.1 tdm-1, error :
    ||=== Paq8px, Release ===|
    D:\Project\Paq8px\Project\Paq8px\main.cpp||In function 'int main(int, char**)':|
    D:\Project\Paq8px\Project\Paq8px\main.cpp|4533|war ning: format '%ld' expects type 'long int', but argument 2 has type 'int'|
    D:\Project\Paq8px\Project\Paq8px\main.cpp|4539|war ning: format '%ld' expects type 'long int', but argument 2 has type 'int'|
    obj\Release\main.o:main.cpp|| undefined reference to 'train'|
    obj\Release\main.o:main.cpp|| undefined reference to 'train'|
    obj\Release\main.o:main.cpp.text$_ZN5Mixer1pEv[Mixer:()]+0x8e)||undefined reference to 'train'|
    obj\Release\main.o:main.cpp.text$_ZN5Mixer1pEv[Mixer:()]+0xed)||undefined reference to 'dot_product'|
    obj\Release\main.o:main.cpp.text$_ZN5Mixer1pEv[Mixer:()]+0x164)||undefined reference to 'dot_product'|
    ||=== Build finished: 5 errors, 2 warnings (0 minutes, 3 seconds) ===|
    any clues?, great compressor btw .

  24. #21
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,908
    Thanks
    291
    Thanked 1,271 Times in 718 Posts
    add "#define NOASM" at start?

  25. #22
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,257
    Thanks
    307
    Thanked 795 Times in 488 Posts
    Or get the assembler code. Any version of paq7 or paq8 will work if it's not included in the latest distribution because the functions train() and dot_product() don't change. All of my versions include them but sometimes people forget. My latest version paq8l at http://mattmahoney.net/dc/#paq should have it.

  26. #23
    Programmer Jan Ondrus's Avatar
    Join Date
    Sep 2008
    Location
    Rychnov nad Kněžnou, Czech Republic
    Posts
    279
    Thanks
    33
    Thanked 138 Times in 50 Posts

    paq8px_v70

    paq8px_70 in this new version i tried to add zlib recompression from AntiZ
    it probably has lot of bugs and can crash randomly
    https://github.com/hxim/paq8px

    EDIT:
    more info:
    a) i think case when recompression is not possible is not handled properly
    b) uses tempfile for recursion so other zlib streams or images/audio/exe can be detected inside inflated data
    c) unlike AntiZ it works in 64kB blocks so there is no need to alocate memory for entire stream data
    Attached Files Attached Files
    Last edited by Jan Ondrus; 9th March 2016 at 08:54. Reason: more info

  27. Thanks (7):

    comp1 (9th March 2016),Darek (9th March 2016),Gonzalo (9th March 2016),kaitz (9th March 2016),Matt Mahoney (11th March 2016),schnaader (9th March 2016),Stephan Busch (9th March 2016)

  28. #24
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    279
    Thanks
    43
    Thanked 50 Times in 40 Posts
    How to add PNG & GIF model to paq8px v70 ? thanx

  29. #25
    Tester
    Stephan Busch's Avatar
    Join Date
    May 2008
    Location
    Bremen, Germany
    Posts
    876
    Thanks
    474
    Thanked 175 Times in 85 Posts
    Dear Jan,

    thank you very much for that new feature in paq8px_v70.
    Can you please also put AntiZ in your FP8?

  30. #26
    Programmer Jan Ondrus's Avatar
    Join Date
    Sep 2008
    Location
    Rychnov nad Kněžnou, Czech Republic
    Posts
    279
    Thanks
    33
    Thanked 138 Times in 50 Posts
    Quote Originally Posted by Stephan Busch View Post
    Dear Jan,

    thank you very much for that new feature in paq8px_v70.
    Can you please also put AntiZ in your FP8?
    Yes, i will probably do it when i have some free time...

  31. Thanks:

    Stephan Busch (9th March 2016)

  32. #27
    Tester
    Stephan Busch's Avatar
    Join Date
    May 2008
    Location
    Bremen, Germany
    Posts
    876
    Thanks
    474
    Thanked 175 Times in 85 Posts
    Quote Originally Posted by Jan Ondrus View Post
    Yes, i will probably do it when i have some free time...
    Thank you very much, Jan.
    I also wrote a PM.

  33. #28
    Programmer Jan Ondrus's Avatar
    Join Date
    Sep 2008
    Location
    Rychnov nad Kněžnou, Czech Republic
    Posts
    279
    Thanks
    33
    Thanked 138 Times in 50 Posts

    paq8px_v71

    fixed bug with special image/audio models broken by zlib recompression
    fixed bug with wrong progress display
    some changes in recompressed zlib data header
    added pdfimage support
    Attached Files Attached Files

  34. Thanks (5):

    kaitz (11th March 2016),Matt Mahoney (11th March 2016),Mike (12th March 2016),schnaader (12th March 2016),Stephan Busch (12th March 2016)

  35. #29
    Member
    Join Date
    Aug 2014
    Location
    Argentina
    Posts
    536
    Thanks
    236
    Thanked 90 Times in 70 Posts
    Quote Originally Posted by Jan Ondrus View Post
    c) unlike AntiZ it works in 64kB blocks so there is no need to alocate memory for entire stream data
    What does that mean? Do you split the original file into 64 kb blocks? Doesn't it break some zlib chunks?

  36. #30
    Programmer Jan Ondrus's Avatar
    Join Date
    Sep 2008
    Location
    Rychnov nad Kněžnou, Czech Republic
    Posts
    279
    Thanks
    33
    Thanked 138 Times in 50 Posts
    Quote Originally Posted by Gonzalo View Post
    What does that mean? Do you split the original file into 64 kb blocks? Doesn't it break some zlib chunks?
    I mean i never pass more than 65536 bytes at time when calling zlib functions inflate() and deflate().
    And also i request at max 65536 bytes by telling output buffer size in avail_out.
    I set avail_in and avail_out in z_stream strucutre <= 65536 (with one exception avail_out can be at max 2*65536 when calling deflate during testing parameters phase because we need to have space for pending data from previous call).
    When testing deflate parameters and finding differences it works like:
    In AntiZ : read input -> call inflate() -> store entire inflated data in memory -> repeat for different deflate params : ( call deflate() -> compare with input )
    In paq8px_v70 : read one block of input -> call inflate on one block -> repeat for different deflate params : (call deflate on one block -> compare with matching data in input block) -> go to next block

    Code:
      for (int i=0; i<len; i+=BLOCK) {
        unsigned int blsize=min(len-i,BLOCK);
        for (int j=0; j<81; j++) {
            if (diffCount[j]>=LIMIT) continue;
            memmove(&zrec[0], &zrec[BLOCK], BLOCK);
            recpos[j]-=BLOCK;
        }
        memmove(&zin[0], &zin[BLOCK], BLOCK);
        fread(&zin[BLOCK], 1, blsize, in); // Read block from input file
        
        // Decompress/inflate block
        main_strm.next_in=&zin[BLOCK]; main_strm.avail_in=blsize;
        do {
            main_strm.next_out=&zout[0]; main_strm.avail_out=BLOCK;
            main_ret=inflate(&main_strm, Z_FINISH);
    
            // Recompress/deflate block with all possible parameters
            for (int j=0; j<81; j++) {
                if (diffCount[j]>=LIMIT) continue;
                rec_strm[j].next_in=&zout[0];  rec_strm[j].avail_in=BLOCK-main_strm.avail_out;
                rec_strm[j].next_out=&zrec[recpos[j]]; rec_strm[j].avail_out=BLOCK*2-recpos[j];
                int ret=deflate(&rec_strm[j], (int)main_strm.total_in == len ? Z_FINISH : Z_NO_FLUSH);
                if (ret!=Z_BUF_ERROR && ret!=Z_STREAM_END && ret!=Z_OK) { diffCount[j]=LIMIT; continue; }
    
                // Compare
                int end=2*BLOCK-(int)rec_strm[j].avail_out;
                int tail=max(main_ret==Z_STREAM_END ? len-(int)rec_strm[j].total_out : 0,0);
                for (int k=recpos[j]; k<end+tail; k++) {
                  if ((k<end && i+k-BLOCK<len && zrec[k]!=zin[k]) || k>=end) {
                    if (++diffCount[j]<LIMIT) {
                      const int p=j*LIMIT+diffCount[j];
                      diffPos[p]=i+k-BLOCK;
                      diffByte[p]=zin[k];
                    }
                  }
                }
                recpos[j]=2*BLOCK-rec_strm[j].avail_out;
            }
        } while (main_strm.avail_out==0 && main_ret==Z_BUF_ERROR);
        if (main_ret!=Z_BUF_ERROR && main_ret!=Z_STREAM_END) break;
      }
    Last edited by Jan Ondrus; 9th March 2016 at 17:09.

  37. Thanks (2):

    Gonzalo (10th March 2016),schnaader (10th March 2016)

Page 1 of 44 12311 ... LastLast

Similar Threads

  1. FrontPAQ - GUI frontend for PAQ8PF and PAQ8PX
    By LovePimple in forum Download Area
    Replies: 26
    Last Post: 17th January 2019, 13:36
  2. Alternative paq8px builds
    By M4ST3R in forum Download Area
    Replies: 20
    Last Post: 25th June 2010, 16:19
  3. Optimized paq7asm.asm code not compatible with paq8px?
    By M4ST3R in forum Data Compression
    Replies: 7
    Last Post: 3rd June 2009, 15:34

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •