Page 21 of 31 FirstFirst ... 111920212223 ... LastLast
Results 601 to 630 of 923

Thread: Paq8pxd dict

  1. #601
    Member
    Join Date
    Jun 2009
    Location
    Puerto Rico
    Posts
    208
    Thanks
    98
    Thanked 27 Times in 20 Posts
    Quote Originally Posted by kaitz View Post
    paq8pxd_v63
    - jpeg model +1 context, use bhmap
    - change wordmodel, textmodel
    Hi. thanks for this new version.

    I think there's a bug when specifying threads.

    In v62 and below, for example, -s15:9 works, but with v63, -s15:9 will attempt to create an archive named -s15 and interprets the archive name as a file, which results in a file not found error. Did something changed here?

    EDIT: Managed to compile it with MinGW and -DMT and now it works
    Last edited by moisesmcardona; 27th January 2019 at 18:24.

  2. #602
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,102
    Thanks
    679
    Thanked 431 Times in 329 Posts
    some enwik scores:
    16'309'641 - enwik8 -s8 by Paq8pxd_v61
    15'968'477 - enwik8 -s15 by Paq8pxd_v61
    16'570'543 - enwik8.drt -s15 by Paq8pxd_v61
    126'432'827 - estimate of enwik9_1423 -s15 by Paq8pxd_v61

    16'567'856 - enwik8.drt -s15 by Paq8pxd_v62

    16'309'012 - enwik8 -s8 by Paq8pxd_v63
    15'967'201 - enwik8 -s15 by Paq8pxd_v63
    16'637'302 - enwik8.drt -s15 by Paq8pxd_v63
    126'422'724 - estimate of enwik9_1423 -s15 by Paq8pxd_v63

    looks like paq8px v63 version scores of enwik are slightly better than v61 and slightly worse than v62. I need to test enwik8 with v62 also.

  3. Thanks:

    kaitz (4th February 2019)

  4. #603
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,102
    Thanks
    679
    Thanked 431 Times in 329 Posts
    I'm on the test of paq8pxd v63 and there is the same issue as in v61 version - for such files form Maximum Compression and Silesia:

    A10.jpg
    FlashMX.pdf
    MSO97.DLL
    ohs.doc

    mozilla
    samba

    and F.JPG from my testset

    with option -s15 program quit w/o any communicate and didn't finish the compression.
    Similar issue was in v61 version, however it didn't occurs in v62.
    I'll try lower settings to these files (v61 works only for -s13 and below).

    Nope -I've tried to compress these files with options -s14 to -s11 and there are the same error -> that's different issue than in v61
    Option -s10 works properly.

    Darek
    Last edited by Darek; 2nd February 2019 at 14:13.

  5. Thanks:

    kaitz (8th February 2019)

  6. #604
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    505
    Thanks
    207
    Thanked 343 Times in 182 Posts
    I probably made mistake somewhere in code when i changed jpeg model.
    KZo


  7. #605
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,102
    Thanks
    679
    Thanked 431 Times in 329 Posts
    enwik9 update for the latest versions:

    16'334'853 - enwik8 -s8 by Paq8pxd_v58
    15'993'302 - enwik8 -s15 by Paq8pxd_v58
    16'607'676 - enwik8.drt -s15 by Paq8pxd_v58
    126'629'383 - enwik9_1423 -s15 by Paq8pxd_v58

    16'309'641 - enwik8 -s8 by Paq8pxd_v61
    15'968'477 - enwik8 -s15 by Paq8pxd_v61
    16'570'543 - enwik8.drt -s15 by Paq8pxd_v61 - small improvement to v58 version = -0.22%
    126'587'796 - enwik9_1423 -s15 by Paq8pxd_v61 - small improvement to v58 version, however smaller than for enwik8 = -0.03%

    16'309'012 - enwik8 -s8 by Paq8pxd_v63 - tiny improvement to v61 version = -0.004%
    15'967'201 - enwik8 -s15 by Paq8pxd_v63 - small improvement to v61 version = -0.01%
    16'637'302 - enwik8.drt -s15 by Paq8pxd_v63
    126'597'584 - enwik9_1423 -s15 by Paq8pxd_v63 - despite enwik8 gains, enwik9 got some little regeression = +0.01%

  8. Thanks:

    kaitz (15th February 2019)

  9. #606
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    505
    Thanks
    207
    Thanked 343 Times in 182 Posts
    Paq8pxd_v64

    • jpeg model fix
    Attached Files Attached Files
    KZo


  10. Thanks:

    Darek (26th February 2019)

  11. #607
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,102
    Thanks
    679
    Thanked 431 Times in 329 Posts
    Updated 4 corpuses scores for paq8pxd v64.
    In total scores are slightly worse than v62 version.
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	paq8pxd_v64_4_Corpuses.jpg 
Views:	72 
Size:	1.25 MB 
ID:	6521  

  12. #608
    Member
    Join Date
    Aug 2008
    Location
    NZ
    Posts
    59
    Thanks
    30
    Thanked 11 Times in 7 Posts
    I'm running Windows 7 SP1 64bit system. Paq8pxd V58 works fine in command line environment, but ALL later versions (from version 60) that I've tried to run do not run! I always get error message that the program has stopped working!?

    What is wrong here?

  13. #609
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,840
    Thanks
    288
    Thanked 1,243 Times in 696 Posts
    Probably "Compiled with AVX2"?
    Here's an SSE2 build using IC19.
    Attached Files Attached Files

  14. Thanks (2):

    brispuss (19th March 2019),xinix (18th March 2019)

  15. #610
    Member
    Join Date
    Aug 2008
    Location
    NZ
    Posts
    59
    Thanks
    30
    Thanked 11 Times in 7 Posts
    Many thanks for providing the SSE2 compile! This build works on my system, which uses an i5-3570K CPU.

    If paq8pxd versions 61 and later use AVX2, then that would explain why the builds do not work on my system as the i5-3570K does NOT support AVX2!

    Although version 64 SSE2 build works, it, unfortunately, produces slightly less compression on a test *.jpg file compared to version 58.

    Original JPG = 3 474 873 bytes
    Compress using paq8pxd version 58 at -s9:4 = 2 711 990 bytes
    Compress using paq8pxd version 64 (SSE2 build) at -s9:4 = 2 712 221 bytes

    Any way to improve compression by maybe using different build??

  16. #611
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,840
    Thanks
    288
    Thanked 1,243 Times in 696 Posts
    I tried compressing a jpeg file with original build of pxd_v64 (AVX2) and mine - the output is identical.
    Sometimes improving compression for one file makes it worse for others, I guess.

    > Any way to improve compression by maybe using different build??

    Try paq8px, or maybe cmix.

  17. #612
    Member
    Join Date
    Aug 2008
    Location
    NZ
    Posts
    59
    Thanks
    30
    Thanked 11 Times in 7 Posts
    Thanks. But paq8px v175 produces slightly worse compression (2 715 638 bytes). And I only have 8 GB of memory, not enough to run cmix (reliably).

    According to post #607, Darek included a chart of compression results for various versions of paq8pxd. The JPG file in that test had slightly better compression using V64 of paq8pxd compared to V58 of paq8pxd. But in my test the compression was slightly worse for a different JPG file. Is this because of different compiles using AVX2 (or whatever) which might produce slightly better compression than compiles using SSE2?

  18. #613
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,102
    Thanks
    679
    Thanked 431 Times in 329 Posts
    In the tests which I posted there only 2 JPG files then it could be not so representative probe for all kind of files.
    @brispuss - If you could send me this file then I could test to compress it by cmix.
    Last edited by Darek; 20th March 2019 at 17:42.

  19. #614
    Member
    Join Date
    Aug 2008
    Location
    NZ
    Posts
    59
    Thanks
    30
    Thanked 11 Times in 7 Posts
    Thanks for the offer of compressing the test JPG file using cmix. But even if cmix compresses the JPG file better than other compressors, I can't use cmix to compress JPG files due to lack of computer memory (only 8GB instead of recommended 32GB+). And I can't afford to upgrade my computer for some time yet, unfortunately. So I must respectfully decline the offer at this time.

    The issue of later versions of paq8pxd (versions 60 and later) not working on my system appear to be related to the compile method used when compiling Windows executables. It is suspected the compiling has been done using AVX2 (or some other method?) which my system can't run as my CPU doesn't support AVX2 (and some other methods?).

    Version 58 of paq8pxd works fine on my system, so presumably that compile used some method that my CPU is compatible with.

    Can perhaps the paq8pxd author (kaitz) consider reverting back to compile methods used on version 58 and earlier versions of paq8pxd, as those compiles are more compatible (for everyone)?
    Last edited by brispuss; 21st March 2019 at 13:43.

  20. #615
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,840
    Thanks
    288
    Thanked 1,243 Times in 696 Posts
    > Can perhaps the paq8pxd author (kaitz) consider reverting back to compile methods used on version 58 and earlier versions of paq8pxd

    There's probably a significant speed difference.

    And in any case, he includes the source, so why can't you compile it?

    1) Download mingw toolchain from https://sourceforge.net/projects/mingw-w64/files/
    Its a simple .7z archive, eg. https://sourceforge.net/projects/min...-rt_v6-rev0.7z
    2) Unpack it to C:\MinGW820x or modify g.bat in attached archive.
    C:\MinGW820x\bin\g++.exe should exist.
    3) Run g.bat, it would work for a while, then produce paq8pxd64.exe
    4) Replace paq8pxd.cpp with any version you'd like to try.
    Attached Files Attached Files

  21. Thanks (2):

    brispuss (25th March 2019),xinix (24th March 2019)

  22. #616
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,102
    Thanks
    679
    Thanked 431 Times in 329 Posts
    Hmmm... on my computer (+Win10 Pro) paq8_v64_avx version hang on always in the same moment:
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	paq8pxd_v64_avx_err.jpg 
Views:	87 
Size:	75.7 KB 
ID:	6524  

  23. #617
    Member
    Join Date
    Aug 2008
    Location
    NZ
    Posts
    59
    Thanks
    30
    Thanked 11 Times in 7 Posts
    Thanks for the details on compiling!

    I thought compiling would be a little easier rather than using batch file(s)?

    The supplied batch file method seems to make compiles OK, so far.

    Actually I did try compiling the cpp files previously using Mingw with g++ but got compile errors in relation to zlib(?)

    I download zlib and set up its path in environment variables, but the compiler still fails to locate the zlib files.

    I'm trying to make compiles using manual input rather than using the more complicated batch file procedure.

    Where should the zlib directory/files be placed so that g++ compiler finds these files?

  24. #618
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,840
    Thanks
    288
    Thanked 1,243 Times in 696 Posts
    > I thought compiling would be a little easier rather than using batch file(s)?

    I think thats easiest.
    The mainstream method relies on make, VS project files, or other build systems,
    which requires installing extra tools and is not compatible with all C++ compilers.
    And would require preparing a less readable kind of script anyway.

    > Actually I did try compiling the cpp files previously using Mingw with g++ but got compile errors in relation to zlib(?)

    Yes, paq8pxd uses zlib to recompress deflate streams.
    That is, it can unpack compressed streams eg. in pdf files,
    and then compress unpacked data with stronger methods.

    > I download zlib and set up its path in environment variables,
    > but the compiler still fails to locate the zlib files.

    gcc/mingw doesn't use environment variables to locate headers.
    Use -I option to specify zlib.h folder location.

    > I'm trying to make compiles using manual input rather than using the more complicated batch file procedure.

    Its really a bad idea, especially in this case.
    Compilers have lots of non-default options which can improve program's performance.
    But the simplest way to compile it is the following:
    1. copy files from zlib folder in my archive to same place as paq8pxd.cpp
    2. make sure that mingw\bin folder is in PATH
    3. Run this: gcc -c *.c && g++ paq8pxd.cpp *.o -DWINDOWS -DMT

    Well, gcc is annoying and can't compile both C and C++ in a single run.
    In theory, single g++ line would be enough after fixing zlib to compile as C++.
    Unfortunately, its not quite valid C++ as is, but that's fixable.

    With VS installed you can also use
    cl.exe -DWINDOWS -DMT *.c*
    But in practice that also is better done with a .bat file.

    > Where should the zlib directory/files be placed so that g++ compiler finds these files?

    Anywhere, with -Ipath added to gcc cmdline.
    But you'd also need to link with zlib too.
    In theory, it can be properly installed in the compiler,
    but for that to work you'd need the whole framework, like unix or cygwin or Msys2.
    zlib is not worth it.

  25. Thanks:

    brispuss (29th April 2019)

  26. #619
    Member
    Join Date
    Aug 2008
    Location
    NZ
    Posts
    59
    Thanks
    30
    Thanked 11 Times in 7 Posts
    Sorry for late reply.

    Thanks again for detailed information!!

    I'm testing several different archivers/compressors to compress some files/software to save space on my HDD.

    Currently compressing this software as a test -

    Code:
    https://www.bigfishgames.com/games/2936/insider-tales-the-stolen-venus/?pc
    Successfully compressed this software using paq8px V170 (after "tarring" the files first into a single .tar file), and also by using Fast paq8 (FP8) v6 as another test.

    Tried compressing the software using paq8pxd V58, and later also tried version 64 (SSE2 compiled version; see post #609). Both versions were run at -s9 compression setting.
    But at around 99.80 ~ 99.90 % completion of compression, paq8pxd stopped and an error message appeared saying that the program (paq8pxd) has stopped and I should close the program! This error occurred on three occasions.

    Next tried paq8pxd V58 at -s4 compression setting, and this time the compression was successful. However, tried to compress using -s5 compression setting, and again got error message appearing at around 99.90% completion of compression.

    So it appears that paq8pxd seems to have some difficulty compressing this particular test software successfully (at higher compression levels)!? Why?

    The test software only consists of around a dozen or so files with a combined total capacity of about 70 MB.

    A reminder that I'm running W7 64 bit, 8GB RAM, and an i5-3570K CPU.

  27. #620
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,840
    Thanks
    288
    Thanked 1,243 Times in 696 Posts
    Could be a bug. Some parts are only enabled at higher compression levels.
    Unfortunately you'd have to wait for kaitz (pxd developer) to fix that.

    Also, it could be a good idea to try precomp first: https://github.com/schnaader/precomp-cpp/releases

  28. Thanks:

    brispuss (30th April 2019)

  29. #621
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    611
    Thanks
    246
    Thanked 240 Times in 119 Posts
    Quote Originally Posted by brispuss View Post
    Code:
    https://www.bigfishgames.com/games/2936/insider-tales-the-stolen-venus/?pc
    Results for the demo version:

    Code:
    Original:         73,547,799 bytes (15 files, 1 folder)
      7-Zip Ultra:      55,819,554 bytes
      7-Zip store:      73,548,286 bytes
        Precomp 0.4.7:  49,297,470 bytes
    Content is not encrypted (which surprised me as BigFish games are often encrypted) and contains PNGs, JPGs and a SWF stream.
    EDIT: Full version seems to be the same, demo version is only time-limited to an hour.
    Last edited by schnaader; 30th April 2019 at 12:25.
    http://schnaader.info
    Damn kids. They're all alike.

  30. Thanks:

    brispuss (30th April 2019)

  31. #622
    Member
    Join Date
    Aug 2008
    Location
    NZ
    Posts
    59
    Thanks
    30
    Thanked 11 Times in 7 Posts
    Thanks for the comments and the brief tests!

    I'm still in the process of trying different compressors/archivers with and without the use of Precomp.

    But at this stage it appears that the paq8 series of compressors do NOT successfully decompress files compressed with paq8 series compressors!

    On one test, an uncompressed executable file of about 30 MB which was compressed using FP8 V6, decompressed to 540 MB!!??

    On another test, a pcf at about 128 MB was compressed using paq8px v170, and on decompression the pcf file was at least 145 MB and still in the process of being decompressed!!??

    So it seems that the paq8 series compressors are unreliable on compressing/decompressing files correctly (losslessly) for files of around 30 MB + and using high compression settings!

    Therefore the paq8 series compressors are no longer being used owing to their apparent unreliability. I'm using other compressors instead for testing in the meantime.

  32. #623
    Member Gotty's Avatar
    Join Date
    Oct 2017
    Location
    Switzerland
    Posts
    403
    Thanks
    278
    Thanked 283 Times in 149 Posts
    Quote Originally Posted by brispuss View Post
    On another test, a pcf at about 128 MB was compressed using paq8px v170, and on decompression the pcf file was at least 145 MB and still in the process of being decompressed!!??
    You may have hit a bug. Please post a downloadable sample to the paq8px thread, so we can investigate where it goes wrong.

  33. #624
    Member
    Join Date
    Aug 2008
    Location
    NZ
    Posts
    59
    Thanks
    30
    Thanked 11 Times in 7 Posts
    I've made a post in that thread. Thanks.

  34. #625
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    505
    Thanks
    207
    Thanked 343 Times in 182 Posts
    paq8pxd_v66
    Code:
    some arch compiler fixes for mmx,ssse3,avx2
    contextMap SIMD change, more speed :D
    matchModel bypass in all streams
    add MJPEG context in jpegModel, use min 256 byte match for bypass
    change BHMap in jpegModel, now does actual compression
    min bypass in image models is width
    in textstream add another mixer context from wordModel
    dmcModel min mem fix
    change APM in EAPM p2
    Code:
    DSCN0791.AVI (30018828 bytes)
                         Compressed  Time    Mem
     paq8pxd_v66.exe -s8 20236432    703.32  2277 MB
     paq8px_v181 -8      20354233    906.67  2378 MB 
     paq8pxd_v64.exe -s8 20558895    668.31  3157 MB
    4_Corpuses_paq8px_v155.jpg (4027286 bytes)
    
    
                         Compressed  Time    Mem
     paq8pxd_v66.exe -s8 2091250    111.43  2086 MB
     paq8px_v181 -8      2119171    136.46  2378 MB
    Compression is in general about same as v54-v55 as i removed some models.

    For jpeg compression bypassing it if long match is found adds to speedup. Helps mostly on artifical images, also on above avi file, ohs.doc, etc.
    Attached Files Attached Files
    KZo


  35. Thanks (3):

    Darek (24th July 2019),Mike (24th July 2019),moisesmcardona (24th July 2019)

  36. #626
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,102
    Thanks
    679
    Thanked 431 Times in 329 Posts
    Scores of my testset for paq8pxd_v66. In total there is 1.24% of loss to v64 version.
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	paq8pxd_v66.jpg 
Views:	46 
Size:	643.3 KB 
ID:	6731  

  37. Thanks:

    kaitz (25th July 2019)

  38. #627
    Member CompressMaster's Avatar
    Join Date
    Jun 2018
    Location
    Lovinobana, Slovakia
    Posts
    184
    Thanks
    49
    Thanked 13 Times in 13 Posts
    @kaitz,
    paq8pxd_v66 stopped working when I have tried to execute it. Printscreen attached.

    Click image for larger version. 

Name:	paq8pxd_v66-error.png 
Views:	70 
Size:	91.4 KB 
ID:	6733

  39. #628
    Member
    Join Date
    Jun 2009
    Location
    Puerto Rico
    Posts
    208
    Thanks
    98
    Thanked 27 Times in 20 Posts
    Quote Originally Posted by CompressMaster View Post
    @kaitz,
    paq8pxd_v66 stopped working when I have tried to execute it. Printscreen attached.

    Click image for larger version. 

Name:	paq8pxd_v66-error.png 
Views:	70 
Size:	91.4 KB 
ID:	6733
    It works in my PC.

  40. #629
    Member CompressMaster's Avatar
    Join Date
    Jun 2018
    Location
    Lovinobana, Slovakia
    Posts
    184
    Thanks
    49
    Thanked 13 Times in 13 Posts
    Quote Originally Posted by moisesmcardona View Post
    It works in my PC.
    Specs?

    Mine:
    Windows 7 Home Premium 64 bit
    Intel Core i3 2310M
    NVIDIA GeForce GT 520M

  41. #630
    Member
    Join Date
    Jun 2009
    Location
    Puerto Rico
    Posts
    208
    Thanks
    98
    Thanked 27 Times in 20 Posts
    Quote Originally Posted by CompressMaster View Post
    Specs?

    Mine:
    Windows 7 Home Premium 64 bit
    Intel Core i3 2310M
    NVIDIA GeForce GT 520M
    AMD Ryzen 7 2700X and 128GB of RAM, Windows 10 Pro 64-bit.

Page 21 of 31 FirstFirst ... 111920212223 ... LastLast

Similar Threads

  1. FreeArc compression suite (4x4, Tornado, REP, Delta, Dict...)
    By Bulat Ziganshin in forum Data Compression
    Replies: 554
    Last Post: 26th September 2018, 02:41
  2. Dict preprocessor
    By pat357 in forum Data Compression
    Replies: 5
    Last Post: 2nd May 2014, 21:51

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •