Page 2 of 3 FirstFirst 123 LastLast
Results 31 to 60 of 88

Thread: PAQ8Q

  1. #31
    Member
    Join Date
    Oct 2007
    Location
    Germany, Hamburg
    Posts
    409
    Thanks
    0
    Thanked 5 Times in 5 Posts
    paq8q_v4.
    Almost only the updated wav model from paq8px_v21. Thanks goes to Jan.

    Although the default speed switch changes from -m5 to -m6. The time shows that people want more then I thought that PAQ remains what it initially was.
    Without an additional parameter -1 to -3 use -m1 (same behavior then in other paq variants) and -4 to -8 use -m6.
    Last edited by Simon Berger; 11th July 2009 at 01:54. Reason: attachment removed

  2. #32
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts

    Question

    Quote Originally Posted by Simon Berger View Post
    paq8q_v4.
    Almost only the updated wav model from paq8px_v21. Thanks goes to Jan.

    Although the default speed switch changes from -m5 to -m6. The time shows that people want more then I thought that PAQ remains what it initially was.
    Without an additional parameter -1 to -3 use -m1 (same behavior then in other paq variants) and -4 to -8 use -m6.
    Thanks Simon!

    Is this paq8q, or have you renamed paq8q2 back to paq8q?

  3. #33
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts

    Thumbs up

    Compiled...

    EDIT: Attachment "paq8q_v4.zip" removed.

    I'm waiting for the test results from my experimental paq8px_v21 SSE2 builds before I release the paq8q_v4 SSE2 builds.

  4. #34
    Member
    Join Date
    Oct 2007
    Location
    Germany, Hamburg
    Posts
    409
    Thanks
    0
    Thanked 5 Times in 5 Posts
    Yes you are right. I renamed it back.

  5. #35
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Sorry, I forgot to replace paq8q.cpp with the new one. This is the correct version...

    EDIT: Attachment "paq8q_v4.zip" removed.

  6. #36
    Member DARcode's Avatar
    Join Date
    May 2009
    Location
    Genoa, Italy
    Posts
    16
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Thanks for the further improvemnets Simon, but the renaming and removal of modes can't all be due to Werner's comments, can they?

    paq8px_speed_optimised v19 -7
    30258245 -> 16874256
    Time 3626.83 sec, used 940594711 bytes of memory

    paq8px_speed_optimised v21 -7
    30258245 -> 16874256
    Time 3543.09 sec, used 940594711 bytes of memory

    paq8q_speed_optimised v4 -7
    30258245 -> 16900586
    Time 2212.72 sec, used 940685239 bytes of memory

  7. #37
    Member
    Join Date
    Oct 2007
    Location
    Germany, Hamburg
    Posts
    409
    Thanks
    0
    Thanked 5 Times in 5 Posts
    Yes it is. I thought before to change the default switch because of more of these comments. I don't understand why but for most guys paq should be the best compression and nothing else. On the other hand many bring the argument paq is much too slow for any usage....
    Okay it's still very slow in faster modes but more then twice as fast.
    The renaming was mostly to follow the style of paq programming. Initially the filename was intended to change every release because then the header is unique and everything is clear but you see how it is now.
    From release of paq8q there were only some small bug fixes that should kept compatibility too. The last change with the wav model was the first that could break it. If I do bigger changes on my own I will at least change it.

    I want to go the way I like but if I can change the program that other guys are more happy without a collision of my own "philosopy" I have absolutely no problem to do so

    EDIT:
    Btw I didn't remove the modes. Only if you don't specify a mode -m6 will be used instead of 5.

  8. #38
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    I changed my mind about waiting for the paq8px tests. Here's some nice SSE2 builds for you to try...

    Please let me know if the new exp02 builds work OK.

    EDIT: Attachments "paq8q_v4_sse2_exp01.zip" and "paq8q_v4_sse2_exp02.zip" removed.

  9. #39
    Member
    Join Date
    Oct 2007
    Location
    Germany, Hamburg
    Posts
    409
    Thanks
    0
    Thanked 5 Times in 5 Posts
    Here is a benchmark for enwik7 (10mb) featuring all 6 modes with sse2b build from exp1. Worked without problems.

    Code:
    # Time percents are added from the last time
    
    original:        10,000,000 bytes     100%     0,0000 sec
    paq8q_v4 -8 -m1:  2,054,669 bytes     20,55%   112,39 sec   + 112,39%
    paq8q_v4 -8 -m2:  2,040,698 bytes     20,41%   121,22 sec   +   7,86%
    paq8q_v4 -8 -m3:  1,952,434 bytes     19,52%   396,06 sec   + 226,73%
    paq8q_v4 -8 -m4:  1,938,205 bytes     19,38%   490,86 sec   +  23,94%
    paq8q_v4 -8 -m5:  1,923,357 bytes     19,23%   583,50 sec   +  18,87%
    paq8q_v4 -8 -m6:  1,920,288 bytes     19,20%  1053,30 sec   +  80,51%
    paq8px_v22 - 8 :  1,919,350 bytes     19,19%  1053,69 sec   +   0,04%

  10. #40
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts

    Thumbs up

    Thanks Simon!

  11. #41
    Member
    Join Date
    Oct 2007
    Location
    Germany, Hamburg
    Posts
    409
    Thanks
    0
    Thanked 5 Times in 5 Posts
    Here is paq8q_v5.

    I included an improved (hope so ) text detection. It is relatively simple but it should be very effective. It can be further improved but this is a first test. I am VERY interested especially in tests with plain ASCII text with special language depended characters. Also some tests with UTF8. I don't think there will be very false detections because of a minimum text length of 80 bytes. But this can be set down without much false detections I guess.

    It based on v4 surely and has no recent improvements of paq8px. As long as paq8px is in such an active development and there is no change to paq8q it makes no sense to put all the new power of paq8px back in this fast.

    NOTE:
    Bear in mind that you need to specify -m2 to -m5 since -m6 is default which has no text detection. -m2 is enough to test the detection.
    Last edited by Simon Berger; 26th May 2009 at 21:09.

  12. #42
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts

    Thumbs up

    Thanks Simon!

    Compiled...

    EDIT: Attachments removed.

  13. #43
    Member
    Join Date
    Oct 2007
    Location
    Germany, Hamburg
    Posts
    409
    Thanks
    0
    Thanked 5 Times in 5 Posts
    Fixed many small bugs that stopped it working right on a german text example with special characters I had built.
    Should be working as expected now.
    Last edited by Simon Berger; 11th July 2009 at 01:54. Reason: attachment removed

  14. #44
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts

    Thumbs up

    Thanks Simon!

    Compiled...

    EDIT: Attachments "paq8q_v6.zip" and "paq8q_v6_sse2.zip" removed.

  15. #45
    Member
    Join Date
    Oct 2007
    Location
    Germany, Hamburg
    Posts
    409
    Thanks
    0
    Thanked 5 Times in 5 Posts
    Here is paq8q_v7.

    Changes in short:

    Code:
    - Fixed compression loss to paq8px caused by splitted sparseModel in -m6 mode
    - Added all improvements of paq8px up to v42
    - Add the CD adresses to the top of the transformation
    - Removed auto Transformation after 16384 sectors
    - Added some information to the documentation part
    I put in all changes from paq8px up to current v42 including cd sector transformation, numerous detection improvements, direct compression without a tempfile of all data except EXE and CD and more.

    But more importantly it closes the gap to paq8px. It has the same (1 byte better because of a changed header) compressed size in the default (-m6) mode.

    I did the first step for compression of bitmaps, audio data etc in CD Bin images by writing the address part of each block to the top of all blocks.
    I would do this in paq8px too, but Jan can decide himself if he like it or not .
    Places of this change are surely only the whole encode_cd and decode_cd.

    Code:
    original: 16,558,080 bytes
    paq8px_v37 -8: 1,832,482
    paq8px_v41 -8: 504,773 bytes
    paq8q_v7 -8: 483,981 bytes
    Nice and surprising big improvement for this high compression level. I will update my other cd test too.
    Last edited by Simon Berger; 11th July 2009 at 01:55. Reason: updated compression example

  16. #46
    Member
    Join Date
    Oct 2007
    Location
    Germany, Hamburg
    Posts
    409
    Thanks
    0
    Thanked 5 Times in 5 Posts
    Paq8q_v8 fixes a bug in decoding of cd images that caused a failed transformation any more then one block in one compression process.

    And here the benchmark of CD.bin

    Code:
    original: 30,359,616 bytes
    paq8px_v37 -8: 14,701,285 bytes
    paq8px_v41 -8: 11,216,893 bytes
    paq8q_v8 -8: 11,150,394 bytes
    No big improvement on this file.

    EDIT:
    CD2.bin with -8 switch.

    Code:
    original: 16,558,080 bytes
    paq8px_v37 -8: 1,832,482
    paq8px_v41 -8: 504,773 bytes
    paq8q_v7 -8: 483,981 bytes
    Last edited by Simon Berger; 11th July 2009 at 01:55. Reason: attachment removed

  17. #47
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts

    Thumbs up

    Thanks Simon!

    Compiled...

    EDIT: Attachments "paq8q_v8.zip" and "paq8q_v8_sse2.zip" removed.

  18. #48
    Programmer Jan Ondrus's Avatar
    Join Date
    Sep 2008
    Location
    Rychnov nad Kněžnou, Czech Republic
    Posts
    279
    Thanks
    33
    Thanked 138 Times in 50 Posts
    Code:
    SUPPORTED FORMATS
    With many detections using the same models there are the following supported
    formats that mostly shows huge compression improvements.
    
    jpegModel:
      - JPEG
    im1bitModel:
      - BMP (1-bit)
    	- TIFF (1-bit; uncomopressed)
    	- PGM (1-bit)
    im8bitModel:
      - BMP (8-bit)
    	- TIFF (8-bit; uncomopressed)
    	- PGM (8-bit)
    	- TGA (normal 8-bit; uncompressed)
    	- TGA (monochrome 8-bit; uncompressed)
    	- RGB (8-bit)
    im24bitModel:
      - BMP (24-bit)
    	- TIFF (24-bit; uncomopressed)
    	- PGM (24-bit)
    	- TGA (normal 24-bit; uncompressed)
    wavModel:
    	- WAVE (1 or 2 channels; 44100 sample rate; 8 bits per sample)
    	- WAVE (1 or 2 channels; 44100 sample rate; 16 bits per sample)
    	- AIFF (1 or 2 channels; 44100 sample rate; 8 bits per sample)
    	- AIFF (1 or 2 channels; 44100 sample rate; 16 bits per sample)
    	- MOD
    	- S3M
    cd:
    	- Transformation of sectors of a CD BIN image
    exeModel:
    	- EXE
    JPEG (only baseline mode)
    S3M (only 8 bits samples)
    PGM (24-bit) -> PPM
    PGM (1-bit) -> PBM
    uncomopressed -> uncompressed
    RGB - also more channels 3 or 4 (= 24 or 32 bits) can be detected - they are compressed using 8-bit image model independently
    WAVE - more sample rates supported
    Last edited by Jan Ondrus; 1st June 2009 at 12:52.

  19. #49
    Programmer Jan Ondrus's Avatar
    Join Date
    Sep 2008
    Location
    Rychnov nad Kněžnou, Czech Republic
    Posts
    279
    Thanks
    33
    Thanked 138 Times in 50 Posts
    Quote Originally Posted by Simon Berger View Post
    Here is paq8q_v7.
    I did the first step for compression of bitmaps, audio data etc in CD Bin images by writing the address part of each block to the top of all blocks.
    I would do this in paq8px too, but Jan can decide himself if he like it or not .
    I have better solution. You don't need to save address part.
    Only first address is saved and every other is computed by adding one to previous one.
    Implemented in paq8px_v43.

  20. #50
    Programmer Jan Ondrus's Avatar
    Join Date
    Sep 2008
    Location
    Rychnov nad Kněžnou, Czech Republic
    Posts
    279
    Thanks
    33
    Thanked 138 Times in 50 Posts
    Quote Originally Posted by Simon Berger View Post
    Paq8q_v8 fixes a bug in decoding of cd images that caused a failed transformation any more then one block in one compression process.
    Does not work for me
    CD image is divided into cd blocks with 7056 bytes (3 sectors).
    Code:
    Creating archive hrady.paq8q with 1 file(s)...
    
    1/1  Filename: hrady (150885504 bytes)
    Block segmentation:
      0 |           cd | 7056 bytes [from 0 to 7055]
      1 |           cd | 7056 bytes [from 7056 to 14111]
      2 |           cd | 7056 bytes [from 14112 to 21167]
      3 |           cd | 7056 bytes [from 21168 to 28223]
      4 |           cd | 7056 bytes [from 28224 to 35279]
      5 |           cd | 7056 bytes [from 35280 to 42335]
      6 |           cd | 7056 bytes [from 42336 to 49391]
      7 |           cd | 7056 bytes [from 49392 to 56447]
      8 |           cd | 7056 bytes [from 56448 to 63503]
      9 |           cd | 7056 bytes [from 63504 to 70559]
     10 |           cd | 7056 bytes [from 70560 to 77615]
     11 |           cd | 7056 bytes [from 77616 to 84671]
     12 |           cd | 7056 bytes [from 84672 to 91727]
     13 |           cd | 7056 bytes [from 91728 to 98783]
     14 |           cd | 7056 bytes [from 98784 to 105839]
     15 |           cd | 7056 bytes [from 105840 to 112895]
     16 |           cd | 7056 bytes [from 112896 to 119951]
     17 |           cd | 7056 bytes [from 119952 to 127007]
     18 |           cd | 7056 bytes [from 127008 to 134063]
     19 |           cd | 7056 bytes [from 134064 to 141119]
     20 |           cd | 7056 bytes [from 141120 to 148175]

  21. #51
    Member
    Join Date
    Oct 2007
    Location
    Germany, Hamburg
    Posts
    409
    Thanks
    0
    Thanked 5 Times in 5 Posts
    Ok that's really a better solution .
    The bug you went in is because of a test in detection I didn't remove. It auto transformed after 3 sectors.

    #paq8q_v9
    Code:
    address transformation from paq8px_v44
    I try to add recursive transformation/detection/compression at the moment. It seems like I found a good method for compression but I might be wrong as long as I didn't get something to work for decoding.
    Last edited by Simon Berger; 11th July 2009 at 01:55. Reason: attachment removed

  22. #52
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts

    Thumbs up

    Thanks Simon!

    Compiled...
    Attached Files Attached Files

  23. #53
    Member
    Join Date
    Oct 2007
    Location
    Germany, Hamburg
    Posts
    409
    Thanks
    0
    Thanked 5 Times in 5 Posts
    Code:
    paq8q_sse2b_intel.exe -8 -m6 out enwik8
    Creating archive out.paq8q with 1 file(s)...
    
    1/1  Filename: enwik8 (100000000 bytes)
    Block segmentation:
      0 |      default | 100000000 bytes [from 0 to 99999999]
    Compressed from 100000000 to 17733026 bytes.
    
    Total 100000000 bytes compressed to 17733056 bytes.
    Time 10677.06 sec, used 1591856848 bytes of memory

  24. #54
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts

    Thumbs up

    Comfortably beating drt|lpaq9 (17964751 bytes) on enwik8.

  25. #55
    Member
    Join Date
    Oct 2007
    Location
    Germany, Hamburg
    Posts
    409
    Thanks
    0
    Thanked 5 Times in 5 Posts
    Here is an experimental paq8q_v10 release.

    It adds recursion compression/decompression to paq8q. But the implementation differs from paq8px. Everything else is on the same functional level.

    Mainly for developers:
    Code:
    - decode function deleted and added decompress + decompressRecursive like in compression part
    - rewrote decode_cd and decode_exe that they can write directly to output file.
    - comparing is done in decode_exe and decompressRecursive
    - 4 bytes more were added for cd blocks containing encoded size
    Please test this with some exe and bin files.

    Known issues:
    Code:
    - decoding percentage display is most likely not right yet
    - code should be able to be still shorter :)

    @Jan please look at this and make your changes as long as it won't be a heavy rewrite on paq8q. I have no problem to port this later to paq8px but I don't want to write here and there too often.
    I sometimes did things that may look not fully optimized to have a short code while performance shouldn't really drop and on other lines that are looking stupid to have no speed drop
    Last edited by Simon Berger; 11th July 2009 at 01:56. Reason: attachment removed

  26. #56
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts

    Thumbs up

    Thanks Simon!

    Compiled...

    EDIT: Attachments "paq8q_v10.zip" and "paq8q_v10_sse2.zip" removed.

  27. #57
    Programmer Jan Ondrus's Avatar
    Join Date
    Sep 2008
    Location
    Rychnov nad Kněžnou, Czech Republic
    Posts
    279
    Thanks
    33
    Thanked 138 Times in 50 Posts
    paq8q_v11
    - some code changed in decompression/compression procedures
    - decompression percentage and skipping files won't work
    Attached Files Attached Files

  28. #58
    Member
    Join Date
    Oct 2007
    Location
    Germany, Hamburg
    Posts
    409
    Thanks
    0
    Thanked 5 Times in 5 Posts
    Yeah decompression percentage is known it should work but it stops earlier. Are you sure skipping won't work? Or only after your changes? I have to say I didn't test it but at least it shouldn't be a big deal to get it to work.

    Ok I know what you mean. Forgot to add something.
    Last edited by Simon Berger; 8th June 2009 at 16:47.

  29. #59
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts

    Thumbs up

    Thanks Simon, and Jan!

    Compiled...

    EDIT: Attachments "paq8q_v11.zip" and "paq8q_v11_sse2.zip" removed.

  30. #60
    Member
    Join Date
    Oct 2007
    Location
    Germany, Hamburg
    Posts
    409
    Thanks
    0
    Thanked 5 Times in 5 Posts
    I like your improvements Jan. Very nice job and so fast . I knew some code wasn't really good but I couldn't see it anymore.

    Here are some improvements. Comparing now works.
    Code:
    paq8q_v12 -d out.paq8q decom
    Extracting 2 file(s) from out.paq8q -5 -m6
    File found. Comparing...
    Block difference found at file offset: 9
    done
    File found. Comparing...
    Files are equal. No difference found.
    Time 15.98 sec, used 281754420 bytes of memory
    I did a change to discarding of files too but still only tested this half.
    Last edited by Simon Berger; 11th July 2009 at 01:56. Reason: attachment removed

Page 2 of 3 FirstFirst 123 LastLast

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •