Page 5 of 22 FirstFirst ... 3456715 ... LastLast
Results 121 to 150 of 642

Thread: Paq8pxd dict

  1. #121
    Member Skymmer's Avatar
    Join Date
    Mar 2009
    Location
    Russia
    Posts
    681
    Thanks
    37
    Thanked 168 Times in 84 Posts
    Quote Originally Posted by Stephan Busch View Post
    File type detection works - but the one wav file of the Audio testset still doesn't get recognized...
    This is the file : http://www.squeezechart.com/rock.wav
    At first I thought that problem is in the file lenght. Rock.wav contains 9553824 samples so considering 44100Hz\2ch\16b the lenght must be (9553824 x 4) + 44 = 38215340 bytes. Its a two bytes less than current size (please note those two bytes). But I was wrong.
    Its appeared that detection problem is caused by non-conforming header. Here is the analysis.

    ChunkIDChunkSizeFormatSubchunk1IDSubchunk1SizeAudioFormatNumChannelsSampleRateByteRateBlockAlignBitsPerSampleExtraParamSizeSubchunk2IDSubchunk2Size
    52494646A61E470257415645666D7420120000000100020044AC000010B1020004001000000064617461801E4702

    Subchunk1Size defines 18 (0x12000000) bytes of subchunk1 lenght. Subchunk1 is the WAVEfmt chunk and in the most cases its value is equal to 16 bytes. Thats why the header has 46 bytes lenght instead of classic 44 bytes.
    Actually there is nothing wrong here because subchunk1 can be larger than 16, but its interesting why it have been expanded. The answer is the ExtraParamSize field. It shouldn't be there because its applicable in case we have non-PCM format. And since AudioFormat field defines 0x0100 (which is PCM) then ExtraParamSize is not needed.
    I don't know if the RIFF specs strictly define the rules of ExtraParamSize field appearence but IMHO any carefull parser should handle such situations.

    Anyway, I have uploaded the modified Rock.wav which is detected as audio by PAQ8pxd_v12. The file is two bytes smaller than original due modified header.
    https://mega.co.nz/#!NctiRSgK!V3WzTD...i3Qsn5UJVM7NPw

  2. The Following User Says Thank You to Skymmer For This Useful Post:

    Stephan Busch (17th August 2014)

  3. #122
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    385
    Thanks
    142
    Thanked 213 Times in 115 Posts
    Quote Originally Posted by Skymmer View Post
    Subchunk1Size defines 18 (0x12000000) bytes of subchunk1 lenght. Subchunk1 is the WAVEfmt chunk and in the most cases its value is equal to 16 bytes. Thats why the header has 46 bytes lenght instead of classic 44 bytes.
    Actually there is nothing wrong here because subchunk1 can be larger than 16, but its interesting why it have been expanded. The answer is the ExtraParamSize field. It shouldn't be there because its applicable in case we have non-PCM format. And since AudioFormat field defines 0x0100 (which is PCM) then ExtraParamSize is not needed.
    I don't know if the RIFF specs strictly define the rules of ExtraParamSize field appearence but IMHO any carefull parser should handle such situations.
    How can I detect whether a WAV file has a 44 or 46-byte header?
    http://stackoverflow.com/questions/1...46-byte-header
    https://ccrma.stanford.edu/courses/4...ts/WaveFormat/

    Quote Originally Posted by AlexDoro View Post
    What is interesting about the "unofficial -9 and -10 compression levels" is that they do not guarantee a better compression ratio. For instance, if you compress the silesia corpus with option -9, you get an overall worse ratio. If you look at files individually, you see that text related ones get compressed better while images and others get compressed worse. I believe that lower memory settings throw away some statistics that affect the performance of the logistic loss. What if -8 level is too high anyway for some of these? What do you think?
    I noticed this to. With wordmodel and nestmodel. If you increase memory you get worse compression. Or no better.
    KZo


  4. The Following User Says Thank You to kaitz For This Useful Post:

    Stephan Busch (17th August 2014)

  5. #123
    Member
    Join Date
    Jun 2014
    Location
    Ro
    Posts
    20
    Thanks
    4
    Thanked 4 Times in 4 Posts

    More testing

    Quote Originally Posted by Sportman View Post
    enwik9 + prog:
    129,827,930 bytes + 422,400 bytes = 130,250,330 bytes (1.63% improvement compared paq8hp12any)
    Sportman, when you have time, will you test the enwik8 and enwik9 with the -10 version of paq8pxd and the files preprocessed with paq8hp12any? It would be nice to see how well it does with a tuned in dictionary.

  6. #124
    Member Skymmer's Avatar
    Join Date
    Mar 2009
    Location
    Russia
    Posts
    681
    Thanks
    37
    Thanked 168 Times in 84 Posts
    There is one important thing I must to say. I had just been noticed that paq8pxd_v12-skbuild-1 is mentioned at Matt's LTCB and it takes the 3d place.
    I don't want me to be taken as a man who reaches some goal by any means necessary by utilizing an ideas of other people. My skbuild was created as an attempt to speed up the paq8pxd_v12 binaries and to include some good ideas from this thread. Yes, I've put some efforts into the skbuild but it would never happened without the core ideas of AlexDoro. Its him who provided the modifications for x64-ready paq8pxd and for -9\-10 levels.
    Please look into these messages:
    http://encode.su/threads/1464-Paq8px...ll=1#post39239
    http://encode.su/threads/1464-Paq8px...ll=1#post39448

    I don't understand how I was so irresponsible and stupid that I didn't mention Alex, so I would like to ask Matt to modify the skbuild description in a way that it will include AlexDoro name.

  7. #125
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 778 Times in 485 Posts
    All the PAQ code including cmix, paq8pxd, paq8hp12, xwrt, lpaq9m, and fp8 are forks of the same project with about 20 contributing authors. Hopefully credit is given in the source code. (zpaq is a complete rewrite). But I will mention AlexDoro in the LTCB description.

  8. #126
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    385
    Thanks
    142
    Thanked 213 Times in 115 Posts
    -option up to 15
    -small changes & fixes
    -all models are classes
    -data streams for diffrent types, independent Encoder and Predictor


    Minimum text detected is 2048 bytes.
    For each stream there is predictor with set of models needed on this type of data.
    All static variables are removed in models.


    In this version data is first analyzed and split into streams:
    Code:
    // Types       Stream
    // DEFAULT HDR 0
    // JPEG        1
    // IMAGE1      2
    // IMAGE4      3
    // IMAGE8      4
    // IMAGE24     5
    // AUDIO       6
    // EXE         7
    // TEXT0       8
    // TXTUTF8 DICTTXT TEXT 9

    8 and 9 are then wrt prerocessed after all data is analyzed. Transform is not tested.
    Then each stream is compressed independetly in a row. This has possibility to use multithreading on streams. Currently not implemented.
    Each stream is stored into temporary file before compression.
    Then predictor is deleted with encoder releasing used memory and another stream compression starts. This allows more memory used for example in IMAGE8 stream witch by default uses -8 about 450MB.


    Option 9 .. 15 are experimental. I cant test them myself.


    Comparison of analyzed data (-0) vm.dll:
    http://encode.su/threads/1464-Paq8px...ll=1#post38948
    Code:
     Segment data size: 1118959 bytes
    
    
     TN |Type name |Count      |Total size
    -----------------------------------------
      0 |default   |     53704 | 2665699007
      1 |jpeg      |       813 |   11514611
      2 |hdr       |       390 |      23405
      3 |1b-image  |       100 |       7344
      4 |4b-image  |        13 |      16288
      5 |8b-image  |        63 |    3083278
      6 |24b-image |       113 |    3690946
      7 |audio     |       101 |    8693627
      8 |exe       |      4467 |  712219218
     10 |text      |     55295 |  539155823
     11 |text0     |      1081 |   35261861
     12 |utf-8     |      7971 |  264789944
     13 |base64    |        21 |      21544
    -----------------------------------------
    Total level  0 |    124132 | 4244176896
    
    
    
    
     TN |Type name |Count      |Total size
    -----------------------------------------
      0 |default   |        21 |      15586
      2 |hdr       |         1 |        118
      4 |4b-image  |         1 |        512
    -----------------------------------------
    Total level  1 |        23 |      16216
    
    
    Compressing default stream.  Total 2665738116
    Compressing jpeg stream.  Total 11514611
    Compressing image1 stream.  Total 7344
    Compressing image4 stream.  Total 16800
    Compressing image8 stream.  Total 3083278
    Compressing image24 stream.  Total 3690946
    Compressing audio stream.  Total 8693627
    Compressing exe stream.  Total 712237086
    Compressing text0 wrt stream.  Total 35261861 wrt num: 32615167
    Compressing text wrt stream.  Total 803945767 wrt: 576665207
    Total 4244176896 bytes compressed to 4015381268 bytes.
    Time 1612.83 sec, used 48.9 MB (51270399 bytes) of memory
    Code:
    DeCompressing default stream.
    DeCompressing jpeg stream.
    DeCompressing image1 stream.
    DeCompressing image1 stream.
    DeCompressing image8 stream.
    DeCompressing image24 stream.
    DeCompressing audio stream.
    DeCompressing exe stream.
    DeCompressing text0 wrt stream.
    DeCompressing text wrt stream.
    Comparing C:\My Backups/vm.dll 4244176896 -> identical
    Time 917.81 sec, used 48.9 MB (51270397 bytes) of memory
    Last edited by kaitz; 26th August 2014 at 06:33. Reason: removed file
    KZo


  9. The Following User Says Thank You to kaitz For This Useful Post:

    schnaader (25th August 2014)

  10. #127
    Member Skymmer's Avatar
    Join Date
    Mar 2009
    Location
    Russia
    Posts
    681
    Thanks
    37
    Thanked 168 Times in 84 Posts
    v13 crashes on my test file at -8 level. Debug-enabled version gives following under debugger:
    Code:
    Assertion failed!
    
    Program: paq8pxd_debug.exe
    File: paq8pxd_v13.cpp, Line 1453
    
    Expression: base+cx<M
    
    This application has requested the Runtime to terminate it in an unusual way.

  11. #128
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    385
    Thanks
    142
    Thanked 213 Times in 115 Posts
    Quote Originally Posted by Skymmer View Post
    v13 crashes on my test file at -8 level. Debug-enabled version gives following under debugger:
    Code:
    Assertion failed!
    
    Program: paq8pxd_debug.exe
    File: paq8pxd_v13.cpp, Line 1453
    
    Expression: base+cx<M
    
    This application has requested the Runtime to terminate it in an unusual way.
    Sorry about that.
    I fixed this.
    Last edited by kaitz; 26th August 2014 at 20:42. Reason: removed file
    KZo


  12. #129
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    778
    Thanks
    63
    Thanked 273 Times in 191 Posts
    Quote Originally Posted by kaitz View Post
    I fixed this.
    -8 work but higher not, while there is enough free memory:

    -9:
    Error: Allocating 536870912 bytes
    Time 0.00 sec, used 1517.8 MB (1591515455 bytes) of memory
    Out of memory

    -10:
    Error: Allocating 2147483648 bytes
    Time 0.00 sec, used 621.2 MB (651407679 bytes) of memory
    Out of memory

    -11:
    Error: Allocating 2147483648 bytes
    Time 0.00 sec, used 941.2 MB (986951999 bytes) of memory
    Out of memory

    -12:
    Error: Allocating 2147483648 bytes
    Time 0.01 sec, used 1581.2 MB (1658040639 bytes) of memory
    Out of memory

    -13:
    Error: Allocating 1073741888 bytes
    Time 0.01 sec, used 1837.2 MB (1926432351 bytes) of memory
    Out of memory

    -14:
    Error: Allocating 2147483648 bytes
    Time 0.00 sec, used 301.2 MB (315813247 bytes) of memory
    Out of memory

    -15:
    Error: Allocating 2147483648 bytes
    Time 0.00 sec, used 301.2 MB (315813247 bytes) of memory
    Out of memory

  13. #130
    Tester
    Stephan Busch's Avatar
    Join Date
    May 2008
    Location
    Bremen, Germany
    Posts
    873
    Thanks
    462
    Thanked 175 Times in 85 Posts
    With the 2 bytes smaller .wav all WAV in the Audio testset are detected correctly but the compressed size is
    396.412.909 bytes where earlier Paq8pxd builds (v2, v3, v4) compressed to about 352.000.000 bytes.
    This entry keeps Paq8pxd_v12 from taking number one position in the chart.

    Paq8pxd_v13 crashes on my system after having shown segmentation of all files in the audio testset.
    So far, it is working on the App testset but compression is not finished yet.
    Last edited by Stephan Busch; 26th August 2014 at 12:52.

  14. #131
    Member Skymmer's Avatar
    Join Date
    Mar 2009
    Location
    Russia
    Posts
    681
    Thanks
    37
    Thanked 168 Times in 84 Posts
    Official binary is 32bit one so it will not run on levels higher than -8. Well, with a little bit of luck and LargeAddressAware flag it can run at -9 but not higher.
    Also paq8pxd_v13fix still crashes on my file.
    Code:
    Program received signal SIGSEGV, Segmentation fault.
    0x0044d97f in wavModel1::p (this=<optimized out>, m=..., info=6, val2=0) at paq8pxd_v13.cpp:3590
    3590      if (level>=4 &&  (rlen>1)) recModel->p(m,rlen);

  15. #132
    Tester
    Stephan Busch's Avatar
    Join Date
    May 2008
    Location
    Bremen, Germany
    Posts
    873
    Thanks
    462
    Thanked 175 Times in 85 Posts
    I used option -8

  16. #133
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    385
    Thanks
    142
    Thanked 213 Times in 115 Posts
    @Stephan - v13 is totally different. But i'm interested in guttenberg and wiki results.
    @Sportman - my compile is only for 32 bit windows.

    Uploaded fix. recModel was not init.
    No change in name.
    Last edited by kaitz; 26th August 2014 at 20:57. Reason: typo
    KZo


  17. #134
    Member
    Join Date
    Jun 2013
    Location
    Sweden
    Posts
    150
    Thanks
    9
    Thanked 25 Times in 23 Posts
    Laptop i7-3610qm 16GB with ramdisk 3GB Win7ult_x64

    Did a quick test (on ramdisk), first ARJ32 a -r -m0 -a1 psp7 psp7\* (paint shop pro 7), then:

    for %a in (1 2 3 4 5 6 7 8 9) do paq8pxd_v13.exe -%a %a-archive psp7.ARJ

    -1 and -2 and -3 worked until 99% then stopped working. I skipped the test after -3 crashed.

    paq8pxd_v13.exe -d 1-archive.paq8pxd13
    Segment data not found. (same with 2 and 3)

    Strange seeing analyzing first, then compression of all methods, but I like it.
    Last edited by a902cd23; 26th August 2014 at 20:27. Reason: added computer info

  18. #135
    Member Skymmer's Avatar
    Join Date
    Mar 2009
    Location
    Russia
    Posts
    681
    Thanks
    37
    Thanked 168 Times in 84 Posts
    We fall into loop )
    Code:
    File: paq8pxd_v13.cpp, Line 1453
    
    Expression: base+cx<M
    
    This application has requested the Runtime to terminate it in an unusual way.

  19. #136
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    385
    Thanks
    142
    Thanked 213 Times in 115 Posts
    Yes, my fault. I will upload correct version and remove others.

    So:
    Attached Files Attached Files
    Last edited by kaitz; 26th August 2014 at 20:57. Reason: file
    KZo


  20. The Following 3 Users Say Thank You to kaitz For This Useful Post:

    Skymmer (27th August 2014),Sportman (27th August 2014),Stephan Busch (27th August 2014)

  21. #137
    Tester
    Stephan Busch's Avatar
    Join Date
    May 2008
    Location
    Bremen, Germany
    Posts
    873
    Thanks
    462
    Thanked 175 Times in 85 Posts
    I like the wy paq8pxd_v13 is going: Analysing and segmenting into groups which could then be compressed
    in parallel - once multithreading support has been enabled. This is great news.

    @Kaido: fix3 works so far.. Testing of the SqueezeChart testsets has begun

  22. #138
    Member Skymmer's Avatar
    Join Date
    Mar 2009
    Location
    Russia
    Posts
    681
    Thanks
    37
    Thanked 168 Times in 84 Posts
    Good news from me too. v13fix3 passes my testset at -8 level without errors. So here is the paq8pxd_v13_skbuild1. It includes speed optimized 32bit and 64bit binaries and original (fix3) sources. Up to 10.7% speedup can be expected.
    Please be carefull with experimental levels cause they require quite a lot of memory.
    Code:
    9	3254 MB
    10	6204 MB
    11	8000 MB
    12	9538 MB
    13	12360 MB
    14	18260 MB
    15	25955 MB
    And the bad news... v13fix3 crashes on my testset at -13\-14\-15 levels. Details later.
    Attached Files Attached Files

  23. The Following 4 Users Say Thank You to Skymmer For This Useful Post:

    kaitz (27th August 2014),Samuraikarte (4th September 2014),Sportman (27th August 2014),Stephan Busch (27th August 2014)

  24. #139
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    778
    Thanks
    63
    Thanked 273 Times in 191 Posts
    paq8pxd_v13_x64 -15:
    enwik8 16,595,606 bytes, 3,004.22 sec.
    enwik9 131,598,576 bytes, 29,924.62 sec.

  25. The Following User Says Thank You to Sportman For This Useful Post:

    Matt Mahoney (28th August 2014)

  26. #140
    Tester
    Stephan Busch's Avatar
    Join Date
    May 2008
    Location
    Bremen, Germany
    Posts
    873
    Thanks
    462
    Thanked 175 Times in 85 Posts
    Paq8pxd_v13fix3 could improve compression if more JPEG inside the camera raw files would be detected.
    It detects 7 JPEG while Precomp detects 44 JPEG (Although I don"t know if the JPEG headers are present and standard).
    According to JPEGsnoop we have a total of 65.228.825 bytes (11.29% of testset) in JPEG format,
    in 10 of 25 RAW the JPEGs are missing EOI-marker. but maybe detection could be improved?

    The Gutenberg testset compressed to 235.217.901 bytes (using option eight) (new world record) - well done Kaido
    The App testset compressed to 60.602.701 bytes (about 1 MB smaller than archive of previous version) - well done
    The Audio testset compressed to 396.480.950 bytes (all wav detected but 50 MB worse than pxd v2, v3 and v4)
    Last edited by Stephan Busch; 28th August 2014 at 15:09.

  27. #141
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 778 Times in 485 Posts
    I added Sportman's results to LTCB. http://mattmahoney.net/dc/text.html#1302
    I noticed that v13 compresses worse than v12 even when using more memory. Is this because it doesn't use a static dictionary?

  28. #142
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    385
    Thanks
    142
    Thanked 213 Times in 115 Posts
    Quote Originally Posted by Stephan Busch View Post
    I like the wy paq8pxd_v13 is going: Analysing and segmenting into groups which could then be compressed
    in parallel - once multithreading support has been enabled. This is great news.
    I have created version with multithreading support. Only compression uses it. It kind of works. Especially if there is more than one datastream.

    Uses level 5 threads 9.
    Code:
    paq8pxd_v14 -5:9 E:\O\PAQ\paq8pxd_v14\rc.tar
    Creating archive E:\O\PAQ\paq8pxd_v14\rc.tar.paq8pxd14 with 1 file(s)...
    
    
    File list (16 bytes)
    Compressed from 16 to 19 bytes.
    
    
    1/1  Filename: E:/O/PAQ/paq8pxd_v14/rc.tar (5410304 bytes)
    Block segmentation:
     0           | default   |       512 b [0 - 511]
     1           | jpeg      |   1601429 b [512 - 1601940]
     2           | default   |       619 b [1601941 - 1602559]
     3           | hdr       |      1078 b [1602560 - 1603637]
     4           | 8b-image  |    995328 b [1603638 - 2598965] (width: 1152)
     5           | default   |       970 b [2598966 - 2599935]
     6           | hdr       |        62 b [2599936 - 2599997]
     7           | 1b-image  |    124416 b [2599998 - 2724413] (width: 144)
     8           | default   |       962 b [2724414 - 2725375]
     9           | hdr       |       118 b [2725376 - 2725493]
     10          | 4b-image  |    497664 b [2725494 - 3223157] (width: 576)
     11          | default   |       906 b [3223158 - 3224063]
     12          | utf-8     |   1000128 b [3224064 - 4224191]
     13          | default   |       832 b [4224192 - 4225023]
     14          | hdr       |        54 b [4225024 - 4225077]
     15          | 24b-image |    786432 b [4225078 - 5011509] (width: 1536)
     16          | default   |       970 b [5011510 - 5012479]
     17          | hdr       |        44 b [5012480 - 5012523]
     18          | audio     |     24247 b [5012524 - 5036770] (8b mono)
     19          | default   |      1872 b [5036771 - 5038642]
     20          | exe       |    283603 b [5038643 - 5322245]
     21          | default   |     88058 b [5322246 - 5410303]
    
    
     Segment data size: 218 bytes
    
    
     TN |Type name |Count      |Total size
    -----------------------------------------
      0 |default   |         9 |      95701
      1 |jpeg      |         1 |    1601429
      2 |hdr       |         5 |       1356
      3 |1b-image  |         1 |     124416
      4 |4b-image  |         1 |     497664
      5 |8b-image  |         1 |     995328
      6 |24b-image |         1 |     786432
      7 |audio     |         1 |      24247
      8 |exe       |         1 |     283603
     12 |utf-8     |         1 |    1000128
    -----------------------------------------
    Total level  0 |        22 |    5410304
    
    
    Compressing default stream.  Total 97057
    Compressing jpeg stream.  Total 1601429
    Compressing image4 stream.  Total 497664
    Compressing image1 stream.  Total 124416
    Compressing image8 stream.  Total 995328
    Compressing exe stream.  Total 283607
    Compressing image24 stream.  Total 786432
    Compressing audio stream.  Total 24247
    Compressing text wrt stream. Total 1000128
     Total 1000128 wrt: 848481
    Total 5410304 bytes compressed to 2428598 bytes.
    Time 181.06 sec, used 1185.4 MB (1242947337 bytes) of memory
    Code:
    DeCompressing default stream.
    DeCompressing jpeg stream.
    DeCompressing image1 stream.
    DeCompressing image4 stream.
    DeCompressing image8 stream.
    DeCompressing image24 stream.
    DeCompressing audio stream.
    DeCompressing exe stream.
    DeCompressing text wrt stream.
    Comparing E:\O\PAQ\paq8pxd_v14/rc.tar 5410304 -> identical
    Time 264.13 sec, used 17592186034005.3 MB (18446744062793171000 bytes) of memory
    Last edited by kaitz; 29th August 2014 at 00:58. Reason: smaple
    KZo


  29. #143
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    385
    Thanks
    142
    Thanked 213 Times in 115 Posts
    paq8pxd_v14

    • multithreading on compression upto 9 threads
    • small changes & fixes



    To use more then one thread use like this:
    paq8pxd14 -4:2 file

    This compresses file using level 4 with 2 threads. /max 9 threads/
    Actual usage of threads depends on number of datastreams. If only on stream then only one thread is used, if more then one stream then max 2 threads is used. And memory usage then is not more then level*numofthreads.
    There is no progress indicator when compressing.

    Decompression is not multithreaded.
    Quote Originally Posted by Stephan Busch View Post
    The Audio testset compressed to 396.480.950 bytes (all wav detected but 50 MB worse than pxd v2, v3 and v4)
    I found out why compression was worse. But with multithreading enabled decompression fails. Without its ok. (MingGW 4.8.1)
    Then i used Visual C++ express 2010 and multithreading enabled compression decompresses ok.
    It has something todo with floating point math, can't figure out why.

    So x86 executable is VC compiled.

    Edit:
    replacing long double with double and compiled with sse math in gcc version makes compression decompression with multithreading identical in wavmodel.
    Still VC vs GCC compiled versions are not compatible when audio compression is used. This has been discussed many times in forum.
    Attached Files Attached Files
    Last edited by kaitz; 3rd September 2014 at 12:38. Reason: float
    KZo


  30. #144
    Member
    Join Date
    Jun 2013
    Location
    Sweden
    Posts
    150
    Thanks
    9
    Thanked 25 Times in 23 Posts
    v14 is not abel to find files is subdir that v13 finds: (same laptop described earlier)

    [E:\ADOBE]e:\paq8pxd_v14.exe -8 CS5 CS5\*.* CS5.md5
    CS5/*.*: not found, skipping...
    Creating archive CS5.paq8pxd14 with 1 file(s)...

    File list (26 bytes)
    Compressed from 26 to 28 bytes.

    1/1 Filename: CS5.md5 (97970 bytes)
    Block segmentation:
    0 | text | 97970 b [0 - 97969]

    Segment data size: 9 bytes

    TN |Type name |Count |Total size
    -----------------------------------------
    10 |text | 1 | 97970
    -----------------------------------------
    Total level 0 | 1 | 97970

    Compressing text wrt stream. Total 97970
    Total 97970 wrt: 77105
    Total 97970 bytes compressed to 18943 bytes.
    Time 5.76 sec, used 1638.0 MB (1717573371 bytes) of memory

    [E:\ADOBE]e:\paq8pxd_v13.exe -8 CS5 CS5\*.* CS5.md5
    Creating archive CS5.paq8pxd13 with 1064 file(s)...

    File list (67738 bytes)
    Compressed from 67738 to 3891 bytes.

    1/1064 Filename: (alot of them...)

  31. #145
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    385
    Thanks
    142
    Thanked 213 Times in 115 Posts
    I rarely use VC. Forgot to add -DWINDOWS.
    Last edited by kaitz; 26th September 2014 at 16:19. Reason: removed link
    KZo


  32. The Following User Says Thank You to kaitz For This Useful Post:

    Stephan Busch (2nd September 2014)

  33. #146
    Tester
    Stephan Busch's Avatar
    Join Date
    May 2008
    Location
    Bremen, Germany
    Posts
    873
    Thanks
    462
    Thanked 175 Times in 85 Posts
    Results of Paq8pxd_v13 (64-bit skbuild) can be found in this log file

    http://www.squeezechart.com/paq8pxd_v13-x64.log

    And here is the log of Paq8pxd_v12 (64-bit skbuild)

    http://www.squeezechart.com/paq8pxd_v12-x64.log

    XML testset: WRT output size is larger and compression of is worse compared to v12
    Last edited by Stephan Busch; 2nd September 2014 at 13:25.

  34. The Following User Says Thank You to Stephan Busch For This Useful Post:

    kaitz (2nd September 2014)

  35. #147
    Member
    Join Date
    Jun 2013
    Location
    Sweden
    Posts
    150
    Thanks
    9
    Thanked 25 Times in 23 Posts
    Testing your latest version:

    [E:\ADOBE]e:\paq8pxd_v14_1.exe -8 CS5 CS5\*.* CS5.md5
    CS5/*.*: not found, skipping...
    Creating archive CS5.paq8pxd14 with 1060 file(s)...

    File list (67623 bytes)
    Compressed from 67623 to 3858 bytes.

    1/1060 Filename: .... and so it continues

    Now I see that it is only 1059 (1060 counting MD5) files that should be compressed. How v13 finds 1064 i dont know. There are folder CS5 (1060 files) and CS6 (2774 files) + both MD5-files in E:\ADOBE.

    And Skymmer´s paq8pxd_v13_x86.exe finds 2119 files there paq8pxd_v13_x64.exe finds 1060.


    I find it very annoying that it copies the data to the% temp% since its on my ramdisk. CS5 is close to 7GB and my ramdisk is only 3GB.
    Last edited by a902cd23; 2nd September 2014 at 20:25.

  36. #148
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    385
    Thanks
    142
    Thanked 213 Times in 115 Posts
    Quote Originally Posted by Matt Mahoney View Post
    I noticed that v13 compresses worse than v12 even when using more memory. Is this because it doesn't use a static dictionary?
    I would also like to know exactly how this result was achieved.

    Quote Originally Posted by Stephan Busch View Post

    http://www.squeezechart.com/paq8pxd_v12-x64.log

    XML testset: WRT output size is larger and compression of is worse compared to v12
    This was expected result. wiki files are multilanguage. And lot of UTF chars are escaped adding to size. Solution is to wrt process block larger then 5-MB ...10MB in another stream separately.
    KZo


  37. #149
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    385
    Thanks
    142
    Thanked 213 Times in 115 Posts
    Quote Originally Posted by a902cd23 View Post
    Now I see that it is only 1059 (1060 counting MD5) files that should be compressed. How v13 finds 1064 i dont know. There are folder CS5 (1060 files) and CS6 (2774 files) + both MD5-files in E:\ADOBE.

    And Skymmer´s paq8pxd_v13_x86.exe finds 2119 files there paq8pxd_v13_x64.exe finds 1060.
    Use -l option on created archive. It will list all files found.
    Quote Originally Posted by a902cd23 View Post
    I find it very annoying that it copies the data to the% temp% since its on my ramdisk. CS5 is close to 7GB and my ramdisk is only 3GB.
    Cant do much about that.
    KZo


  38. #150
    Tester
    Stephan Busch's Avatar
    Join Date
    May 2008
    Location
    Bremen, Germany
    Posts
    873
    Thanks
    462
    Thanked 175 Times in 85 Posts
    paq8pxd_14 exits saying "Error: Allocating 536870976 bytes" when using -8:2 and over 6 GB free memory..
    Maybe a 64-bit build would be useful here.

    Would there be a way to avoid those wav model problems? Maybe a solution without floating point math?
    I would also vote for a fast version of paq8pxd_v14 - like MCM.

Page 5 of 22 FirstFirst ... 3456715 ... LastLast

Similar Threads

  1. FreeArc compression suite (4x4, Tornado, REP, Delta, Dict...)
    By Bulat Ziganshin in forum Data Compression
    Replies: 554
    Last Post: 26th September 2018, 02:41
  2. Dict preprocessor
    By pat357 in forum Data Compression
    Replies: 5
    Last Post: 2nd May 2014, 21:51

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •