Page 7 of 22 FirstFirst ... 5678917 ... LastLast
Results 181 to 210 of 642

Thread: Paq8pxd dict

  1. #181
    Member Skymmer's Avatar
    Join Date
    Mar 2009
    Location
    Russia
    Posts
    681
    Thanks
    37
    Thanked 168 Times in 84 Posts
    Please look here.
    Quote Originally Posted by kaitz View Post
    Last version has problems in level 1-3. There is no compression.
    Version v14 itself has some problems and it was stated that compression doesn't work on 1-3 levels. Probably its also true for -0. Also filtering mode is not guaranteed.
    Testing must be done on 4 to 8 levels.
    Last edited by Skymmer; 28th September 2014 at 21:39.

  2. #182
    Member
    Join Date
    May 2012
    Location
    United States
    Posts
    323
    Thanks
    178
    Thanked 52 Times in 37 Posts
    Quote Originally Posted by Skymmer View Post
    Please look here.

    Version v14 itself has some problems and it was stated that compression doesn't work on 1-3 levels. Probably its also true for -0. Also filtering mode is not guaranteed.
    Testing must be done on 4 to 8 levels.
    Ahh ok thank you. -0 works great on v15 as a preprocessor for many other compressors so I immediately assumed and tried -0 on your modified v14 version.

    Thanks for pointing out.

  3. #183
    Member
    Join Date
    May 2008
    Location
    brazil
    Posts
    163
    Thanks
    0
    Thanked 3 Times in 3 Posts
    Quote Originally Posted by comp1 View Post
    Ahh ok thank you. -0 works great on v15 as a preprocessor for many other compressors so I immediately assumed and tried -0 on your modified v14 version.

    Thanks for pointing out.
    Good idea. I think preprocessing modes should assume negative values (like -1 for light preprocessing and -10 for heavy preprocessing) . Zero should always be the good old "store".
    Last edited by lunaris; 29th September 2014 at 03:51.

  4. #184
    Member
    Join Date
    May 2012
    Location
    United States
    Posts
    323
    Thanks
    178
    Thanked 52 Times in 37 Posts
    Quote Originally Posted by lunaris View Post
    Good idea. I think preprocessing modes should assume negative values (like -1 for light preprocessing and -10 for heavy preprocessing) . Zero should always be the good old "store".
    Well actually that is not what I meant

    I put the "-" in my previous post to indicate the syntax that is used now (or which I use) in paq8pxd.

    paq8pxd_vxx -0 file.ext

    "-0" is store but it does it's exe, audio, text, etc. preprocessing. This helps many other programs like UDA, LPAQ, etc.

    II did not mean "-0" as a "negative". If I was compressing with level 8, I would would type "-8" because of the program's syntax, not positive, nor negative.

    But YOUR (not mine) idea is not a bad one.

  5. #185
    Member
    Join Date
    May 2008
    Location
    brazil
    Posts
    163
    Thanks
    0
    Thanked 3 Times in 3 Posts
    Quote Originally Posted by comp1 View Post
    Well actually that is not what I meant

    I put the "-" in my previous post to indicate the syntax that is used now (or which I use) in paq8pxd.

    paq8pxd_vxx -0 file.ext

    "-0" is store but it does it's exe, audio, text, etc. preprocessing. This helps many other programs like UDA, LPAQ, etc.

    II did not mean "-0" as a "negative". If I was compressing with level 8, I would would type "-8" because of the program's syntax, not positive, nor negative.

    But YOUR (not mine) idea is not a bad one.

    I understand what you say. My idea is use negative numbers for preprocessing because thery are the reverse of compression . like --1
    Last edited by lunaris; 29th September 2014 at 04:45.

  6. #186
    Member
    Join Date
    May 2012
    Location
    United States
    Posts
    323
    Thanks
    178
    Thanked 52 Times in 37 Posts
    Quote Originally Posted by lunaris View Post
    I understand what you say. My ideia is use negative numbers for preprocessing because thery are the reverse of compression . like --1
    Got ya Good idea.

  7. #187
    Member
    Join Date
    Jun 2013
    Location
    Sweden
    Posts
    150
    Thanks
    9
    Thanked 25 Times in 23 Posts

    10 days of testing PXD v15

    BSC and LZT was run on i7-3770k with 32GB 2400mhz 1.5TB-USB2 and RAID 2x1TB, while PXD15 was run on laptop i7-3610QM 16GB 1600mhz SSD 240GB + 320GB-USB2. Both files was pxd15:ed comp/decomp simultanious on USB2.

    4 011 655 168 bytes C1.iso
    3 390 994 760 bytes lzturbo v1.2 (-49 -b1024, less than 1hour)
    3 249 606 836 bytes bsc v1.2 x64 (491.856 seconds -s -r -b800 -ca -e2 -G -M255 -H12)
    3 180 909 545 bytes paq8pxd_15 -s8 (338155.46 sec)

    4 695 719 936 bytes A2.iso
    4 331 977 945 bytes lzturbo v1.2 (-49 -b1024, less than 1hour)
    4 203 000 930 bytes bsc v1.2 x64 (475.319 seconds -s -r -b800 -ca -e2 -G -M255 -H12)
    4 174 920 917 bytes paq8pxd_15 -s8 (395889.83 sec)

    Both 7z and winrar was slightly larger than lzt.

    ----------------------------------------------------------------------------------------------

    paq8pxd_v15.exe -s8
    Creating archive C1.iso.paq8pxd15 with 1 file(s)...

    File list (199 bytes)
    Compressed from 199 to 139 bytes.

    1/1 Filename: C1.iso (4011655168 bytes)
    Block segmentation:
    0 | default | 26272358 b [0 - 26272357]
    1 | jpeg | 73535 b [26272358 - 26345892]
    2 | default | 125196789 b [26345893 - 151542681]
    3 | jpeg | 194001 b [151542682 - 151736682]
    4 | default |1417606367 b [151736683 - 1569343049]
    5 | jpeg | 79548 b [1569343050 - 1569422597]
    6 | default |1708087908 b [1569422598 - 3277510505]
    7 | jpeg | 124646 b [3277510506 - 3277635151]
    8 | default | 456376306 b [3277635152 - 3734011457]
    9 | jpeg | 144721 b [3734011458 - 3734156178]
    10 | default | 277498989 b [3734156179 - 4011655167]

    Segment data size: 99 bytes

    TN |Type name |Count |Total size
    -----------------------------------------
    0 |default | 6 | 4011038717
    1 |jpeg | 5 | 616451
    -----------------------------------------
    Total level 0 | 11 | 4011655168

    Compressing default stream. Total 4011038717
    Compressing jpeg stream. Total 616451
    Total 4011655168 bytes compressed to 3180909545 bytes.
    Time 338155.46 sec, used 1629.6 MB (1708769229 bytes) of memory


    paq8pxd_v15.exe -d C1.iso.paq8pxd15
    DeCompressing default stream.
    DeCompressing jpeg stream.
    Extracting ./C1.iso 4011655168 -> done
    Time 339682.49 sec, used 1629.6 MB (1708769713 bytes) of memory

    CRC32 checked and OK.

    ----------------------------------------------------------------------------------------------

    paq8pxd_v15.exe -s8
    Creating archive A2.iso.paq8pxd15 with 1 file(s)...

    File list (76 bytes)
    Compressed from 76 to 72 bytes.

    1/1 Filename: A2.iso (4695719936 bytes)
    Block segmentation:
    0 | default |4695719936 b [0 - 4695719935]

    Segment data size: 9 bytes

    TN |Type name |Count |Total size
    -----------------------------------------
    0 |default | 1 | 4695719936
    -----------------------------------------
    Total level 0 | 1 | 4695719936

    Compressing default stream. Total 4695719936
    Total 4695719936 bytes compressed to 4174920917 bytes.
    Time 395889.83 sec, used 1629.6 MB (1708769352 bytes) of memory


    paq8pxd_v15.exe -d A2.iso.paq8pxd15
    DeCompressing default stream.
    Extracting ./A2.iso 4695719936 -> done
    Time 394249.75 sec, used 1629.6 MB (1708769496 bytes) of memory

    CRC32 checked and OK.

  8. #188
    Member
    Join Date
    Sep 2014
    Location
    United States
    Posts
    5
    Thanks
    0
    Thanked 0 Times in 0 Posts
    In addition to the erroneous out of memory issue on levels above s8 and f9, I'm have another error that I haven't seen before. I am trying to compress an old software package that runs MechWarrior 2 in DosBox. Here is the command line input and output

    Code:
    C:\Users\Thomas\SkyDrive\Documents\MechVM-win32-2011-04-04c-original>paq8pxd_v15.exe -s8 "MechVM-win32-2011-04-04c"
    Creating archive MechVM-win32-2011-04-04c.paq8pxd15 with 1676 file(s)...
    
    
    File list (106593 bytes)
    Compressed from 106593 to 5662 bytes.
    
    
    1/1676  Filename: MechVM-win32-2011-04-04c/campaigns/campaign-CJF-enh.xml (9151 bytes)
    Block segmentation:
     0           | text      |      9151 b [0 - 9150]
    
    
    2/1676  Filename: MechVM-win32-2011-04-04c/campaigns/campaign-CJF.xml (8876 bytes)
    Block segmentation:
     0           | text      |      8876 b [0 - 8875]
    
    
    3/1676  Filename: MechVM-win32-2011-04-04c/campaigns/campaign-gbl.xml (7377 bytes)
    Block segmentation:
     0           | text      |      7377 b [0 - 7376]
    
    
    4/1676  Filename: MechVM-win32-2011-04-04c/campaigns/campaign-MechVM.xml (1802 bytes)
    Block segmentation:
     0           | text      |      1802 b [0 - 1801]
    
    
    5/1676  Filename: MechVM-win32-2011-04-04c/campaigns/campaign-mercs.xml (7002 bytes)
    Block segmentation:
     0           | text      |      7002 b [0 - 7001]
    
    
    6/1676  Filename: MechVM-win32-2011-04-04c/campaigns/campaign-NetMechLab.xml (2048 bytes)
    Block segmentation:
     0           | text      |      2048 b [0 - 2047]
    
    
    7/1676  Filename: MechVM-win32-2011-04-04c/campaigns/campaign-WC-enh.xml (6087 bytes)
    Block segmentation:
     0           | text      |      6087 b [0 - 6086]
    
    
    8/1676  Filename: MechVM-win32-2011-04-04c/campaigns/campaign-WC.xml (9363 bytes)
    Block segmentation:
     0           | text      |      9363 b [0 - 9362]
    
    
    9/1676  Filename: MechVM-win32-2011-04-04c/campaigns/netmech-campaign.xml (15734 bytes)
    Block segmentation:
     0           | text      |     15734 b [0 - 15733]
    
    
    10/1676  Filename: MechVM-win32-2011-04-04c/campaigns/USERSTAR3.BWD (352 bytes)
    Block segmentation:
     0           | default   |       352 b [0 - 351]
    
    
    11/1676  Filename: MechVM-win32-2011-04-04c/deutsch.lang (1065 bytes)
    Block segmentation:
     0           | text      |      1065 b [0 - 1064]
    
    
    12/1676  Filename: MechVM-win32-2011-04-04c/DOSBox-0.74/Documentation/AUTHORS.txt (276 bytes)
    Block segmentation:
     0           | text      |       276 b [0 - 275]
    
    
    13/1676  Filename: MechVM-win32-2011-04-04c/DOSBox-0.74/Documentation/COPYING.txt (18332 bytes)
    Block segmentation:
     0           | text      |     18332 b [0 - 18331]
    
    
    14/1676  Filename: MechVM-win32-2011-04-04c/DOSBox-0.74/Documentation/INSTALL.txt (3937 bytes)
    Block segmentation:
     0           | text      |      3937 b [0 - 3936]
    
    
    15/1676  Filename: MechVM-win32-2011-04-04c/DOSBox-0.74/Documentation/NEWS.txt (32130 bytes)
    Block segmentation:
     0           | text      |     32130 b [0 - 32129]
    
    
    16/1676  Filename: MechVM-win32-2011-04-04c/DOSBox-0.74/Documentation/README.txt (63204 bytes)
    Block segmentation:
     0           | text      |     63204 b [0 - 63203]
    
    
    17/1676  Filename: MechVM-win32-2011-04-04c/DOSBox-0.74/Documentation/THANKS.txt (1001 bytes)
    Block segmentation:
     0           | default   |      1001 b [0 - 1000]
    
    
    18/1676  Filename: MechVM-win32-2011-04-04c/DOSBox-0.74/DOSBox 0.74 Manual.txt (63204 bytes)
    Block segmentation:
     0           | text      |     63204 b [0 - 63203]
    
    
    19/1676  Filename: MechVM-win32-2011-04-04c/DOSBox-0.74/DOSBox 0.74 Options.bat (107 bytes)
    Block segmentation:
     0           | text      |       107 b [0 - 106]
    
    
    20/1676  Filename: MechVM-win32-2011-04-04c/DOSBox-0.74/dosbox-0.74.conf (11023 bytes)
    Block segmentation:
     0           | text      |     11023 b [0 - 11022]
    
    
    21/1676  Filename: MechVM-win32-2011-04-04c/DOSBox-0.74/DosBox.bat (35 bytes)
    Block segmentation:
     0           | text      |        35 b [0 - 34]
    
    
    22/1676  Filename: MechVM-win32-2011-04-04c/DOSBox-0.74/DOSBox.exe (3727360 bytes)
    Block segmentation:
     0           | default   |      1064 b [0 - 1063]
     1           | exe       |   2998633 b [1064 - 2999696]
    Detect wav sr 0
    C:\Users\Thomas\SkyDrive\Documents\MechVM-win32-2011-04-04c-original>
    As you can see, it is failing after trying to compress DOSBox.exe. This also occurs on level f9.

    The next file I think it was about to include in the archive was a .conf file (just a text file).

  9. #189
    Member
    Join Date
    Sep 2014
    Location
    United States
    Posts
    5
    Thanks
    0
    Thanked 0 Times in 0 Posts
    I went ahead and tried with rounding. The following is a screenshot of the results. Is it normal for compression level 10 to be the smallest, rather than 15? The file sizes in the ls command reflect various compression levels of msvc.exe as packaged by you using mingw64.exe in the rounding-math folder. Notice that compression level 10 is the smallest at 53306 bytes while compression level 15 is the largest among the experimental compression levels at 53714 bytes. I have not yet tried decompressing.
    Click image for larger version. 

Name:	Capture.PNG 
Views:	268 
Size:	50.6 KB 
ID:	3173

  10. #190
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    385
    Thanks
    142
    Thanked 213 Times in 115 Posts
    Quote Originally Posted by freelancer91 View Post

    22/1676 Filename: MechVM-win32-2011-04-04c/DOSBox-0.74/DOSBox.exe (3727360 bytes)
    Block segmentation:
    0 | default | 1064 b [0 - 1063]
    1 | exe | 2998633 b [1064 - 2999696]
    Detect wav sr 0
    C:\Users\Thomas\SkyDrive\Documents\MechVM-win32-2011-04-04c-original>
    [/CODE]

    As you can see, it is failing after trying to compress DOSBox.exe. This also occurs on level f9.

    The next file I think it was about to include in the archive was a .conf file (just a text file).
    It happens in detection phase.
    http://encode.su/threads/1464-Paq8px...ll=1#post40589

    I will upload new version next week.
    KZo


  11. #191
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    385
    Thanks
    142
    Thanked 213 Times in 115 Posts
    -changed tta filter to v2.0
    -display progress with streamid in -DMT (de/compression) overlaps with other threads
    +quick mode
    -more adaptive 8bit image model
    -change text predictor
    -fix wav detect,szdd
    Attached Files Attached Files
    KZo


  12. The Following User Says Thank You to kaitz For This Useful Post:

    surfersat (24th October 2014)

  13. #192
    Member
    Join Date
    Sep 2014
    Location
    Italy
    Posts
    26
    Thanks
    33
    Thanked 18 Times in 11 Posts

    Talking paq8pxd_v12_biondivers1_x64

    Trying to improve compression i made some changes to paq8pxd_v12 to enable -11 level compression.

    My results:

    ENWIK8:

    Compressing enwik8 file with -11 option i get a size of 16,361,221 bytes
    ("paq8pxd_v12-skbuild -10" in Matt Mahoney's LTCB is 16,372,331 bytes)
    Decompression is ok.

    SILESIA:

    For silesia i get a size of 34,708,883 bytes
    (Testing standard "paq8pxd_v12 -8" i obtain a 34,914,762 bytes file size:In "Silesia Open source Compression Benchmark" is listed at 34,693,843 but i don't know why)
    Decompression not yet tested

    ==Usage==

    paq8pxd_v12_biondivers1_x64 -11 nomefile

    ==Note==

    A) Don't specify any compression level to use -11 as default level
    B) -11 compression level require ca. 7.5 gb of RAM
    C) Executable is compiled for 64 bit and corei7 architecture. ask me for other architetures or compile source...


    Could someone test on enwik9?


    ---ENWIK8 COMPRESSION------------

    C:\Bin\paq8pxd_v12-lbversion-1>paq8pxd_v12-prove -11 c:\bin\testset\enwik8
    Creating archive c:\bin\testset\enwik8.paq8pxd12 with 1 file(s)...

    File list (18 bytes)
    Compressed from 18 to 17 bytes.

    1/1 Filename: c:/bin/testset/enwik8 (100000000 bytes)
    Block segmentation:
    0 | utf-8 | 100000000 b [0 - 99999999](wt: 61838435)
    Compressed from 100000000 to 16361168 bytes.

    Total 100000000 bytes compressed to 16361221 bytes.

    Segment data size: 9 bytes

    TN |Type name |Count |Total size
    -----------------------------------------
    11 |utf-8 | 1 | 100000000
    -----------------------------------------
    Total level 0 | 1 | 100000000

    Time 13589.10 sec, used 384393326 bytes of memory



    ---ENWIK8 DECOMPRESSION-------------

    C:\Bin\paq8pxd_v12-lbversion-1>paq8pxd_v12-prove -d c:\bin\testset\enwik8.paq8px
    d12
    Extracting c:\bin\testset/enwik8 100000000 -> done
    Time 11189.71 sec, used 384391300 bytes of memory


    ---SILESIA COMPRESSION--------------

    C:\Bin\paq8pxd_v12-lbversion-1>paq8pxd_v12-prove -11 c:\bin\testset\silesia
    Creating archive c:\bin\testset\silesia.paq8pxd12 with 12 file(s)...

    File list (271 bytes)
    Compressed from 271 to 123 bytes.

    1/12 Filename: c:/bin/testset/silesia/dickens (10192446 bytes)
    Block segmentation:
    0 | text | 10192446 b [0 - 10192445](wt: 6006940)
    Compressed from 10192446 to 1955245 bytes.

    2/12 Filename: c:/bin/testset/silesia/mozilla (51220480 bytes)
    Block segmentation:
    0 | default | 16003634 b [0 - 16003633]
    1 | text | 565152 b [16003634 - 16568785](wt: 378889)
    2 | default | 33443184 b [16568786 - 50011969]
    3 | utf-8 | 575462 b [50011970 - 50587431](wt: 469165)
    4 | default | 51416 b [50587432 - 50638847]
    5 | jpeg | 9407 b [50638848 - 50648254]
    6 | default | 833 b [50648255 - 50649087]
    7 | jpeg | 49629 b [50649088 - 50698716]
    8 | default | 547 b [50698717 - 50699263]
    9 | hdr | 44 b [50699264 - 50699307]
    10 | audio | 27760 b [50699308 - 50727067] (8b mono)
    11 | default | 493412 b [50727068 - 51220479]
    Compressed from 51220480 to 10114721 bytes.

    3/12 Filename: c:/bin/testset/silesia/mr (9970564 bytes)
    Block segmentation:
    0 | default | 9970564 b [0 - 9970563]
    Compressed from 9970564 to 2060964 bytes.

    4/12 Filename: c:/bin/testset/silesia/nci (33553445 bytes)
    Block segmentation:
    0 | text | 33553445 b [0 - 33553444](wt: 30644691)
    Compressed from 33553445 to 847835 bytes.

    5/12 Filename: c:/bin/testset/silesia/ooffice (6152192 bytes)
    Block segmentation:
    0 | default | 4228 b [0 - 4227]
    1 | exe | 5012819 b [4228 - 5017046]
    2 | default | 26830 b [5017047 - 5043876]
    3 | exe | 253183 b [5043877 - 5297059]
    4 | default | 855132 b [5297060 - 6152191]
    Compressed from 6152192 to 1426284 bytes.

    6/12 Filename: c:/bin/testset/silesia/osdb (10085684 bytes)
    Block segmentation:
    0 | default | 10085684 b [0 - 10085683]
    Compressed from 10085684 to 2089157 bytes.

    7/12 Filename: c:/bin/testset/silesia/reymont (6627202 bytes)
    Block segmentation:
    0 | text | 6501239 b [0 - 6501238](wt: 5389241)
    1 | default | 125963 b [6501239 - 6627201]
    Compressed from 6627202 to 809682 bytes.

    8/12 Filename: c:/bin/testset/silesia/samba (21606400 bytes)
    Block segmentation:
    0 | default | 279004 b [0 - 279003]
    1 | text | 1658664 b [279004 - 1937667](wt: 1040875)
    2 | default | 131757 b [1937668 - 2069424]
    3 | text | 2661772 b [2069425 - 4731196](wt: 1699537)
    4 | default | 1092855 b [4731197 - 5824051]
    5 | text | 725004 b [5824052 - 6549055](wt: 562109)
    6 | default | 420432 b [6549056 - 6969487]
    7 | jpeg | 8020 b [6969488 - 6977507]
    8 | default | 461300 b [6977508 - 7438807]
    9 | text | 678792 b [7438808 - 8117599](wt: 554037)
    10 | default | 9673 b [8117600 - 8127272]
    11 | text | 13132289 b [8127273 - 21259561](wt: 9306794)
    12 | default | 346838 b [21259562 - 21606399]
    Compressed from 21606400 to 2854067 bytes.

    9/12 Filename: c:/bin/testset/silesia/sao (7251944 bytes)
    Block segmentation:
    0 | default | 7251944 b [0 - 7251943]
    Compressed from 7251944 to 3791338 bytes.

    10/12 Filename: c:/bin/testset/silesia/webster (41458703 bytes)
    Block segmentation:
    0 | text | 41458703 b [0 - 41458702](wt: 29889927)
    Compressed from 41458703 to 4903261 bytes.

    11/12 Filename: c:/bin/testset/silesia/x-ray (8474240 bytes)
    Block segmentation:
    0 | default | 8474240 b [0 - 8474239]
    Compressed from 8474240 to 3592119 bytes.

    12/12 Filename: c:/bin/testset/silesia/xml (5345280 bytes)
    Block segmentation:
    0 | text | 5345279 b [0 - 5345278](wt: 3560684)
    1 | default | 1 b [5345279 - 5345279]
    Compressed from 5345280 to 263687 bytes.

    Total 211938580 bytes compressed to 34708883 bytes.

    Segment data size: 373 bytes

    TN |Type name |Count |Total size
    -----------------------------------------
    0 |default | 22 | 89529471
    1 |jpeg | 3 | 67056
    2 |hdr | 1 | 44
    7 |audio | 1 | 27760
    8 |exe | 2 | 5266002
    10 |text | 11 | 116472785
    11 |utf-8 | 1 | 575462
    -----------------------------------------
    Total level 0 | 41 | 211938580

    Time 24412.42 sec, used 1199023185 bytes of memory
    Attached Files Attached Files

  14. #193
    Member BetaTester's Avatar
    Join Date
    Dec 2010
    Location
    Brazil
    Posts
    43
    Thanks
    0
    Thanked 3 Times in 3 Posts
    Modify line 6173 in paq8pxd_v16.cpp to
    #include "ttaenc3.cpp" //v2.0

    (paq8pxd_v16.cpp into file paq8pxd_v16.7z submited by Kaitz)

    ===Compile===

    D:\PAQ\bin>"x86_64-w64-mingw32-g++.exe" -march=corei7-avx -O3 -c D:\PAQ\paq8pxd_v16.cpp -o D:\PAQ\paq8pxd_v16.o
    D:\PAQ\bin>"x86_64-w64-mingw32-g++.exe" -static -lstdc++ -dynamic -o D:\PAQ\paq8pxd_v16.exe D:\PAQ\paq8pxd_v16.o

    ===Compile===


    ===TEST===
    D:\PAQ>paq8pxd_v16.exe -f15 anyFile.txt
    Creating archive anyFile.txt.paq8pxd16 with 1 file(s)...

    File list (15 bytes)
    Compressed from 15 to 18 bytes.

    1/1 Filename: anyFile.txt (0 bytes)
    Block segmentation:

    Segment data size: 0 bytes

    TN |Type name |Count |Total size
    -----------------------------------------
    -----------------------------------------
    Total level 0 | 0 | 0

    Total 0 bytes compressed to 133 bytes.
    Time 0.05 sec, used 25893.6 MB (27151418664 bytes) of memory
    ===TEST===

    OK! Paq8pxd_v16.exe works in Win7sp1 x64 / i7-4970 / 32GB RAM / x86_64-w64-mingw32-g++.exe (GCC) 5.0.0 20140918 (experimental)
    Attached Files Attached Files

  15. #194
    Member Skymmer's Avatar
    Join Date
    Mar 2009
    Location
    Russia
    Posts
    681
    Thanks
    37
    Thanked 168 Times in 84 Posts
    Quote Originally Posted by LucaBiondi View Post
    Trying to improve compression i made some changes to paq8pxd_v12 to enable -11 level compression.
    Nice trial man, but you forgot to mention that you copied and pasted the skbuild source and sources of versions that are higher than v12.
    Also your build is dynamic so you need libgcc_s_sjlj-1.dll and libstdc++-6.dll
    And this means that you're losing speed by utilizing not the optimal exception handler for Win platform.

    Quote Originally Posted by BetaTester View Post
    Modify line 6173 in paq8pxd_v16.cpp to
    #include "ttaenc3.cpp" //v2.0

    (paq8pxd_v16.cpp into file paq8pxd_v16.7z submited by Kaitz)
    Another hero. It seems that today is the paq day )
    There is no ttaenc3.cpp in this release. And its not needed for successfull compiling.

    Quote Originally Posted by BetaTester View Post
    D:\PAQ\bin>"x86_64-w64-mingw32-g++.exe" -march=corei7-avx -O3 -c D:\PAQ\paq8pxd_v16.cpp -o D:\PAQ\paq8pxd_v16.o
    Its a very bad idea to use -march=corei7-avx cause it will not run on CPUs which lack of AVX instructions. I cant even run it on my i7-2700.
    Last edited by Skymmer; 19th October 2014 at 13:18.

  16. #195
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 660 Times in 354 Posts
    i7-2700 has the full AVX support

  17. #196
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    925
    Thanks
    544
    Thanked 363 Times in 271 Posts
    Quote Originally Posted by BetaTester View Post
    Modify line 6173 in paq8pxd_v16.cpp to
    #include "ttaenc3.cpp" //v2.0

    (paq8pxd_v16.cpp into file paq8pxd_v16.7z submited by Kaitz)

    ===Compile===

    D:\PAQ\bin>"x86_64-w64-mingw32-g++.exe" -march=corei7-avx -O3 -c D:\PAQ\paq8pxd_v16.cpp -o D:\PAQ\paq8pxd_v16.o
    D:\PAQ\bin>"x86_64-w64-mingw32-g++.exe" -static -lstdc++ -dynamic -o D:\PAQ\paq8pxd_v16.exe D:\PAQ\paq8pxd_v16.o

    ===Compile===


    ===TEST===
    D:\PAQ>paq8pxd_v16.exe -f15 anyFile.txt
    Creating archive anyFile.txt.paq8pxd16 with 1 file(s)...

    File list (15 bytes)
    Compressed from 15 to 18 bytes.

    1/1 Filename: anyFile.txt (0 bytes)
    Block segmentation:

    Segment data size: 0 bytes

    TN |Type name |Count |Total size
    -----------------------------------------
    -----------------------------------------
    Total level 0 | 0 | 0

    Total 0 bytes compressed to 133 bytes.
    Time 0.05 sec, used 25893.6 MB (27151418664 bytes) of memory
    ===TEST===

    OK! Paq8pxd_v16.exe works in Win7sp1 x64 / i7-4970 / 32GB RAM / x86_64-w64-mingw32-g++.exe (GCC) 5.0.0 20140918 (experimental)
    Thank you for this compile.

    It works generally fine with Win7, however, there is something strange with wav compression compared to Kaitz compile. My examples:

    Your compile:

    1/1 Filename: 0.wav (2762044 bytes)
    Block segmentation:
    0 | hdr | 44 b [0 - 43]
    1 | audio | 2762000 b [44 - 2762043] (8b mono)

    Segment data size: 31 bytes

    TN |Type name |Count |Total size
    -----------------------------------------
    2 |hdr | 1 | 44
    7 |audio | 1 | 2762000
    -----------------------------------------
    Total level 0 | 2 | 2762044

    Compressing default stream(0). Total 44
    Compressing audio stream(6). Total 11048000

    Total 2762044 bytes compressed to 1375815 bytes.
    Time 39.51 sec, used 1629.6 MB (1708771003 bytes) of memory


    and Kaitz compile:

    1/1 Filename: 0.WAV (2762044 bytes)
    Block segmentation:
    0 | hdr | 44 b [0 - 43]
    1 | audio | 2762000 b [44 - 2762043] (8b mono)

    Segment data size: 31 bytes

    TN |Type name |Count |Total size
    -----------------------------------------
    2 |hdr | 1 | 44
    7 |audio | 1 | 2762000
    -----------------------------------------
    Total level 0 | 2 | 2762044

    Compressing default stream(0). Total 44
    Compressing audio stream(6). Total 11048000

    Total 2762044 bytes compressed to 1355015 bytes.
    Time 48.81 sec, used 1629.6 MB (1708769403 bytes) of memory

    Other, non-wave files are comressed ok, and had the same size as from Kaitz version.
    Darek

  18. #197
    Member BetaTester's Avatar
    Join Date
    Dec 2010
    Location
    Brazil
    Posts
    43
    Thanks
    0
    Thanked 3 Times in 3 Posts
    Paq8pxd_v16 enumerator to NESROM, line 914
    typedef enum {DEFAULT,  JPEG, HDR, IMAGE1,IMAGE4, IMAGE8, IMAGE24, AUDIO, EXE, CD, TEXT,TEXT0,  TXTUTF8,NESROM, BASE64, SZDD,DICTTXT,BIGTEXT} Filetype;


    implemented in lines 5392
        // NES rom
    if (buf0==0x4E45531A) nesh=i,nesp=0;
    if (nesh) {
    const int p=int(i-nesh);
    if (p==1) nesp=buf0&0xff; //count of pages*0x3FFF
    else if (p==6 && (buf0&0xfe!=0) )nesh=0;
    else if (p==11 && (buf0!=0) )nesh=0;
    else if (p==12) {
    if (nesp>1 && nesp<129) NES_DET(NESROM,nesh-3,15,nesp*0x3FFF);
    nesh=0;
    }
    }


    but is commented in other places:
    line 6417
    //else if (type==NESROM) return 11;

    line 6434
    //else if ((type==NESROM) && streamid==11) return true;


    ===TEST NESROM====

    D:\PAQ>paq8pxd_v16.exe -s0 "Mario Bros. (U) [!].nes"
    Creating archive Mario Bros. (U) [!].nes.paq8pxd16 with 1 file(s)...

    File list (31 bytes)
    Compressed from 31 to 36 bytes.

    1/1 Filename: Mario Bros. (U) [!].nes (24592 bytes)
    Block segmentation:
    0 | default | 24592 b [0 - 24591]

    Segment data size: 9 bytes

    TN |Type name |Count |Total size
    -----------------------------------------
    0 |default | 1 | 24592
    -----------------------------------------
    Total level 0 | 1 | 24592

    Compressing default stream(0). Total 24592
    Total 24592 bytes compressed to 24748 bytes.
    Time 0.03 sec, used 39.4 MB (41346528 bytes) of memory

  19. #198
    Member
    Join Date
    Sep 2014
    Location
    Italy
    Posts
    26
    Thanks
    33
    Thanked 18 Times in 11 Posts

    paq8pxd_v12_biondivers1_x64 (static compile)

    Yes i started from skbuild version of paq8pxd_v12 then i try to add -11 compression level.
    Thanks very much Skymmer for your hint about static compile: I'm really starting to learn both c++ language and paq8pxd logic.
    This is the static compiled exe

    p.s. i compile using the syntax: "g++ paq8pxd_v12_biondivers1_x64.cpp -static -DWINDOWS -DNOASM -Ofast -msse2 -s -march=corei7 -fpermissive -o paq8pxd_v12_biondivers1_x64.exe" but i'm not sure if i use the best options...

    Summary:

    Name Original size Size
    --------------------------------------------
    Calgary.tar 3.152.896 588.933 (*)
    enwik7 10.000.000 1.828.231 (*)
    enwik8 100.000.000 16.361.221 (*)
    silesia 211.938.580 34.708.883 (*)
    enwik9 1.000.000.000 129.435.477 (*) (**)

    (*) paq8pxd_v12_biondivers1_x64 -11
    (**) tested by Sportman
    Attached Files Attached Files
    Last edited by LucaBiondi; 29th October 2014 at 01:26.

  20. #199
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    385
    Thanks
    142
    Thanked 213 Times in 115 Posts
    Quote Originally Posted by Darek View Post

    It works generally fine with Win7, however, there is something strange with wav compression compared to Kaitz compile.
    Looks like tta v3.4.1 vs v2.0


    KZo


  21. #200
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    925
    Thanks
    544
    Thanked 363 Times in 271 Posts
    Quote Originally Posted by kaitz View Post
    Looks like tta v3.4.1 vs v2.0


    It may be this issue.

    There is another problem with this compile V16. Maybe it's connected to memory usage: my computer have 16GB RAM and in modes s13, s14 and s15 program crashes on JPG compression. Only.
    Other data types are compressed well, without any obstacles and despites the s15 mode consume 25GB of RAM it works (it probably use swap file to do this).

    Due to this crash I cannot test Silesia benchmark in modes s13-s15 because Mozilla and Samba files contains JPEG blocks

    My scores of Silesia test compressed by paq8pxd_V16 (BetaTester compile):

    compression mode: s11
    TOTAL SILESIA: 34 550 441
    dickens 1 966 050
    mozilla 10 119 632
    mr 2 053 312
    mci 843 483
    ooffice 1 404 350
    osdb 2 059 876
    reymont 801 487
    samba 2 797 547
    sao 3 776 748
    webster 4 872 573
    x-ray 3 589 030
    xml 266 353


    compression mode: s12
    TOTAL SILESIA: 34 540 013
    dickens 1 965 968
    mozilla 10 111 657
    mr 2 053 278
    mci 842 415
    ooffice 1 404 167
    osdb 2 059 567
    reymont 801 428
    samba 2 797 460
    sao 3 776 652
    webster 4 872 086
    x-ray 3 589 016
    xml 266 319

    It means that in both modes s11 and s12 this release is better than latest best paq version: paq8pxd_v15 -s9..

    Darek

  22. #201
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    385
    Thanks
    142
    Thanked 213 Times in 115 Posts
    On line 3063 (paq8pxd_v16.cpp) there is ... t(CMlimit(MEM())) ...
    Should be t(CMlimit(MEM())/9) or t(CMlimit(MEM())/(level>12?9:1))
    This limits array to 2GB.
    KZo


  23. #202
    Member Skymmer's Avatar
    Join Date
    Mar 2009
    Location
    Russia
    Posts
    681
    Thanks
    37
    Thanked 168 Times in 84 Posts
    I want to ask a question and I believe that I'm not the only one who wants to get an answer to it.
    The question is: what was the reason to drop the OptimFrog based audio model and substitute it with weaker TTA ?
    I like the PAQ family projects and its development in general, I respect the right of programmer to choose the way that his project goes. But with all of my affection to PAQ I just can't understand the reason of cutting off the Optimfrog model.
    Obviously Optimfrog based model is stronger than TTA and works on any sample rates while TTA supports a limited subset of standard sample rates. Yes, at some point there were problems with compiling in order to get identical results on x86\x64 compiles and MSVC compiles, but maybe its gone unnoticed that I found the way to solve it.
    Also I have a suspicion that usage of the MSVC compile as some kind of reference was not so correct. If you'll look into OptimFrog's SDK then you'll see that there are some pragmas used for MSVC compiler.
    Code:
    OptimFROG.h
    #if defined(CFG_MSCVER_WIN32_MIX86)
    #pragma pack(push)
    #pragma pack(1)
    #endif
    Maybe its the solution for long double and the floating point math, maybe its workaround for something else. I don't claim anything and just want to show that OptimFrog's author utilizes some tricks.
    So maybe its better to return to OptimFrog and not lose the compression strenght instead of stepping back with TTA?

    As for the current version. Here is the pack with both x86 and x64 speed optimized compiles. Builded with latest fix for array. Not too much optimizations made but anyway.
    BetaMaster's x64 compile crashes on my WinXP x64 so the resulting table is kinda uninformative.
    Tested at -s6:7 on my TestBed2_small
    Code:
    native		betamaster	skbuild
    132.765s	-		127.906s
    -		crash		122.250s
    Attached Files Attached Files

  24. #203
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    385
    Thanks
    142
    Thanked 213 Times in 115 Posts
    Quote Originally Posted by Skymmer View Post
    I want to ask a question and I believe that I'm not the only one who wants to get an answer to it.
    The question is: what was the reason to drop the OptimFrog based audio model and substitute it with weaker TTA ?
    To remove floating point math from models and speed. /what about other compilers/ To see what else can be used.etc.
    Quote Originally Posted by Skymmer View Post
    Yes i know.
    Quote Originally Posted by Skymmer View Post
    So maybe its better to return to OptimFrog and not lose the compression strenght instead of stepping back with TTA?
    Depends, anyone can make there own version.
    Quote Originally Posted by Skymmer View Post
    As for the current version. Here is the pack with both x86 and x64 speed optimized compiles. Builded with latest fix for array. Not too much optimizations made but anyway.
    BetaMaster's x64 compile crashes on my WinXP x64 so the resulting table is kinda uninformative.
    Tested at -s6:7 on my TestBed2_small
    Code:
    native        betamaster    skbuild
    132.765s    -        127.906s
    -        crash        122.250s
    BetaTester? This version uses another filter (v3.4.1 used in paq8pxd_v15)

    And i will not release any new version. No time. Maybe next year, or year after that. I have reached most of my goals in last version.
    Things that seemed impossible to me many years ago.
    I'm not really a programmer. And i believe anyone can make there own version if they want.

    KZo


  25. #204
    Member
    Join Date
    Aug 2014
    Location
    Argentina
    Posts
    468
    Thanks
    203
    Thanked 81 Times in 61 Posts
    Hi! Maybe I've arrived here too late. But... How about a MP2 model?

    http://encode.su/threads/515-unpackmp2-a-preprocessor-for-lossless-compression-programs


    v1.2 (2010-04-02)
    preserve non-audio data before first and after last mp2 frame (e.g. ID3 tags).
    preserve non-audio data inside of mp2 frames.

    [...]

    This new version of the program also stores non-audio data from the mp2 files. This means that original and transformed mp2 files are identical bit for bit (except for bugs...), which can be verified using tools like fc or diff.


    Attached Files


    GPL V3. One file in C, 556 lines.


    Very fast (~2.5 mb/s on my old machine).

    Expected gain ~12 to ~15 % over the same algorithm applied to mp2 without preprocessing, according to my records. - NOT an exhaustive test.

    Skymmer?

    BTW: I'd love to see it as part of FreeArc.
    Last edited by Gonzalo; 27th October 2014 at 02:47. Reason: Stats.

  26. The Following User Says Thank You to Gonzalo For This Useful Post:

    Bulat Ziganshin (27th October 2014)

  27. #205
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 660 Times in 354 Posts
    i have not seen any mp2 files, so only way to see it in freearc is to someoine else implementing it as cls dll

  28. #206
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    778
    Thanks
    63
    Thanked 273 Times in 191 Posts
    Enwik8:
    16,590,184 bytes, 2,985.408 sec., paq8pxd_v16_x64_skbuild -s15

    Enwik9:
    131,504,841 bytes, 29,883.864 sec. paq8pxd_v16_x64_skbuild -s15

  29. #207
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    385
    Thanks
    142
    Thanked 213 Times in 115 Posts
    KZo


  30. The Following User Says Thank You to kaitz For This Useful Post:

    Stephan Busch (28th October 2014)

  31. #208
    Member
    Join Date
    Sep 2014
    Location
    Italy
    Posts
    26
    Thanks
    33
    Thanked 18 Times in 11 Posts
    Hi Sportman,
    I would test paq8pxd_v12_biondivers1_x64.7z with enwik9 but i have not enough ram on my laptop.
    I you like, could you test it? Otherwise do not worry!
    The syntax is paq8pxd_v12_biondivers1_x64.7z -11 enwik9.

    bye and thanks
    Luca
    Last edited by LucaBiondi; 28th October 2014 at 01:16.

  32. #209
    Member
    Join Date
    Aug 2014
    Location
    Argentina
    Posts
    468
    Thanks
    203
    Thanked 81 Times in 61 Posts
    Quote Originally Posted by Bulat Ziganshin
    i have not seen any mp2 files, so only way to see it in freearc is to someoine else implementing it as cls dll
    Ok. Seems fair. Although this is a bit off-topic, I think it worth to point that even when is true there are not proper *.mp2 files around hard drives, the mp2 codec is widely used in video containers and DVDs. Of course, in order to handle these streams we need to build a parser or a blind search engine like precomp.
    Imagine a DVD image or folder (actually, thousands of them) that can't be compressed by any other archiver behind a 99.8% BUT your software happily shrinks it to a 90%... With less than 600 lines of code.
    Last edited by Gonzalo; 28th October 2014 at 03:10.

  33. #210
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    733
    Thanked 660 Times in 354 Posts
    i hope it will be YOUR compressor

  34. The Following User Says Thank You to Bulat Ziganshin For This Useful Post:

    Gonzalo (28th October 2014)

Page 7 of 22 FirstFirst ... 5678917 ... LastLast

Similar Threads

  1. FreeArc compression suite (4x4, Tornado, REP, Delta, Dict...)
    By Bulat Ziganshin in forum Data Compression
    Replies: 554
    Last Post: 26th September 2018, 02:41
  2. Dict preprocessor
    By pat357 in forum Data Compression
    Replies: 5
    Last Post: 2nd May 2014, 21:51

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •