Page 35 of 36 FirstFirst ... 2533343536 LastLast
Results 1,021 to 1,050 of 1056

Thread: Paq8pxd dict

  1. #1021
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,272
    Thanks
    802
    Thanked 545 Times in 415 Posts
    Quote Originally Posted by kaitz View Post
    Code:
    15642246 - enwik8 -x15 -w -e1,english.dic.txt  paq8pxd_v93 14069.03 sec
    @kaitz - which english.dic file are you used?

  2. #1022
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,272
    Thanks
    802
    Thanked 545 Times in 415 Posts
    Scores of 4 Corpuses for paq8pxd_v93. The best overall scores for all 4 tests and majority of files!
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	paq8pxd_v93_4_Corpuses.jpg 
Views:	37 
Size:	2.93 MB 
ID:	8222  

  3. Thanks:

    LucaBiondi (1st January 2021)

  4. #1023
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,272
    Thanks
    802
    Thanked 545 Times in 415 Posts
    @kaitz - happy New Year and wish the next will be better!

  5. Thanks (4):

    kaitz (1st January 2021),LucaBiondi (1st January 2021),schnaader (8th January 2021),xinix (1st January 2021)

  6. #1024
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    538
    Thanks
    225
    Thanked 392 Times in 203 Posts
    paq8pxd_v94
    Code:
    -change jpeg model
    -fix/change detection (tiff,text)

    Code:
    paq8pxd_v93  -s8 mill.jpg 7132151 4933064  215.93 sec
    paq8pxd_v94  -s8 mill.jpg 7132151 4928860  222.67 sec
    Attached Files Attached Files
    KZo


  7. Thanks (4):

    Darek (8th January 2021),Mike (8th January 2021),moisesmcardona (8th January 2021),xinix (7th January 2021)

  8. #1025
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,272
    Thanks
    802
    Thanked 545 Times in 415 Posts
    Scores of my testset for paq8pxd v94. For F.JPG file there is 215 bytes of gain. Other files remains the same.
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	paq8pxd_v94_DBA.jpg 
Views:	15 
Size:	852.6 KB 
ID:	8253  

  9. Thanks:

    kaitz (13th January 2021)

  10. #1026
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    538
    Thanks
    225
    Thanked 392 Times in 203 Posts
    paq8pxd_v95
    Code:
    jpeg model:
             -more context in Map1  (20)
             -more inputs from main context
             -2 main mixer inputs + 1 apm
             -cleanup
    Code:
                                                    Size Compressed   Sec
    paq8pxd_v95 -s8 a10.jpg                       842468     618555    43.42 sec 1984 MB
    paq8px_v200 -8  a10.jpg                       842468     624597    26.51 sec 2602 MB
    paq8pxd_v95 -s8 mill.jpg                     7132151    4910289   350.38 sec 1984 MB
    paq8px_v200 -8  mill.jpg                     7132151    4952115   228.65 sec 2602 MB
    paq8pxd_v95 -s8 paq8px_v193_4_Corpuses.jpg   3340610    1367528   167.13 sec 1984 MB
    paq8px_v200 -8  paq8px_v193_4_Corpuses.jpg   3340610    1513850   105.90 sec 2602 MB
    paq8pxd_v95 -s8 DSCN0791.AVI                30018828   19858827  1336.94 sec 1984 MB
    paq8px_v200 -8  DSCN0791.AVI                30018828   20171981   992.85 sec 2602 MB
    So mill.jpg is 18571 bytes better v95 vs v94.
    Quote Originally Posted by kaitz View Post
    Code:
                                                      Size  Compressed  Sec
    paq8pxd_v90 -s8  A10.jpg                         842468    623059    25
    paq8px -8        A10.jpg                         842468    624651    36
    paq8pxd_v90 -s8  paq8px_v193_4_Corpuses.jpg     3340610   1418082    96
    paq8px -8        paq8px_v193_4_Corpuses.jpg     3340610   1513474   142
    Quote Originally Posted by kaitz View Post

    Code:
    paq8pxd_v93  -s8 mill.jpg 7132151 4933064  215.93 sec
    paq8pxd_v94  -s8 mill.jpg 7132151 4928860  222.67 sec
    Its slower, im sure nobody cares. Some main context changes have 0 time penalty but improve result some kb.
    For a10.jpg new Map1 context add only about 5 sec.
    Attached Files Attached Files
    KZo


  11. Thanks (3):

    Darek (14th January 2021),Mike (13th January 2021),moisesmcardona (14th January 2021)

  12. #1027
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    514
    Thanks
    63
    Thanked 96 Times in 75 Posts
    Where can I get mill.jpg and dscn0791.avi files? Thank you.. could you upload them here please ?

  13. #1028
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    538
    Thanks
    225
    Thanked 392 Times in 203 Posts
    No
    KZo


  14. #1029
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,272
    Thanks
    802
    Thanked 545 Times in 415 Posts
    Scores of my testset and $ Corpuses for paq8pxd v94 and paq8pxd v95.
    Good improvements on JPG files and files contains such structures.
    A10.JPG file got 618'527 bytes!
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	paq8pxd_v95_4_Corpuses.jpg 
Views:	13 
Size:	3.10 MB 
ID:	8278   Click image for larger version. 

Name:	paq8pxd_v95_DBA_Corpus.jpg 
Views:	9 
Size:	852.0 KB 
ID:	8279  

  15. Thanks:

    kaitz (17th January 2021)

  16. #1030
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,272
    Thanks
    802
    Thanked 545 Times in 415 Posts
    Scores of paq8pxd v95 and previous versions for enwik8 and enwik9:

    15'654'151 - enwik8 -x15 -w -e1,english.dic by paq8pxd_v89_60_4095, change: 0,00%, time 10422,07s
    122'945'119 - enwik9 -x15 -w -e1,english.dic by paq8pxd_v89_60_4095, change: -0,06%, time 100755,31s - best score for paq8pxd versions

    15'647'580 - enwik8 -x15 -w -e1,english.dic by paq8pxd_v90, change: -0,04%, time 9670,5s
    123'196'527 - enwik9 -x15 -w -e1,english.dic by paq8pxd_v90, change: 0,20%, time 110200,16s

    15'642'246 - enwik8 -x15 -w -e1,english.dic by paq8pxd_v95, change: 0,00%, time 10130s- best score for paq8pxd versions (the same as paq8pxd v93 and paq8pxd v94)
    123'151'008 - enwik9 -x15 -w -e1,english.dic by paq8pxd_v95, change: 0,00%, time 102009,55s

  17. Thanks:

    kaitz (18th January 2021)

  18. #1031
    Member
    Join Date
    Jun 2009
    Location
    Puerto Rico
    Posts
    277
    Thanks
    164
    Thanked 64 Times in 49 Posts
    @kaitz, I opened another MR https://github.com/kaitz/paq8pxd/pull/15.

    Since you added a BZip2 transform, the BZip2 library needed to be added to the CMakeLists so that it can detect it and allow to compile your latest versions

  19. #1032
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    538
    Thanks
    225
    Thanked 392 Times in 203 Posts
    paq8pxd_v99
    Code:
    -xml/html like content processed separately in wordmodel
    -adjust some wordmodel parameters
    -attempt to detect ISO latin text
    -some changes in detection (cleanup)
    1 thing helps on silesia webster/xml
    2 helps 1bit on most text files (1k loss on enwik8)
    3 helps on silesia samba maybe mozilla and any other text file prev detected as bintext
    4 in tar mode some files are treated as text by extension without confirming (.c,.h,.html,.cpp,.po,.txt), just a bit faster processing. (for examp, linux kernel source tar)
    Attached Files Attached Files
    KZo


  20. Thanks (4):

    Darek (24th January 2021),Mike (21st January 2021),moisesmcardona (21st January 2021),xinix (22nd January 2021)

  21. #1033
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    538
    Thanks
    225
    Thanked 392 Times in 203 Posts
    Silesia:
    Click image for larger version. 

Name:	px201.vs.pxd99.png 
Views:	33 
Size:	4.2 KB 
ID:	8288
    KZo


  22. Thanks:

    Darek (22nd January 2021)

  23. #1034
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    538
    Thanks
    225
    Thanked 392 Times in 203 Posts
    paq8pxd_v100
    Code:
    add lstm model back (active on -x option)used on all predictors exept audio
    add matchModel from paq8px_v201 as second model
    adjust old matchModel parameters
    tar header as bintext
    add back 1 mixer context if DECA
    in sparsemodel (default) add 2 contexts
    adjust normalModel
    Fixes https://encode.su/threads/1464-Paq8p...ll=1#post66290
    Attached Files Attached Files
    KZo


  24. Thanks (5):

    Darek (24th January 2021),Mauro Vezzosi (25th January 2021),Mike (24th January 2021),moisesmcardona (24th January 2021),xinix (24th January 2021)

  25. #1035
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,272
    Thanks
    802
    Thanked 545 Times in 415 Posts
    Scores for paq8pxd v99 and paq8pxd v100 for my testset.
    Paq8pxd v99 version is about 15KB better than paq8px v95.
    paq8pxd v100 version is abput 35KB better than paq8px v99 - mainly due to lstm implementation, however it's not as big gain like in paq8px version (about 95KB between non-lstm and lstm versions)
    Timings for paq8pxd v99, paq8pxd v100 and paq8px v200 (-l) versions:
    paq8pxd v99 = 5'460,32s
    paq8pxd v100 = 11'610,80s = 2.1 times slower - it's still about 1.7 times faster than paq8px
    paq8px v200 = 19'440,11s
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	paq8pxd_v99_DBA_Corpus.jpg 
Views:	14 
Size:	854.4 KB 
ID:	8292   Click image for larger version. 

Name:	paq8pxd_v100_DBA_Corpus.jpg 
Views:	15 
Size:	855.9 KB 
ID:	8293  

  26. Thanks:

    kaitz (25th January 2021)

  27. #1036
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    514
    Thanks
    63
    Thanked 96 Times in 75 Posts
    by using paq8sk44 -s8 option on f.jpg (DBA corpus) the result is
    Total 112038 bytes compressed to 80194 bytes.
    Time 19.17 sec, used 2444 MB (2563212985 bytes) of memory

  28. Thanks:

    Darek (26th January 2021)

  29. #1037
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    538
    Thanks
    225
    Thanked 392 Times in 203 Posts
    silesia
    Code:
    paq8pxd      v99     v100
                 -s8      -s8   diff
    dickens  1895705  1895269    436
    mozilla  6917463  6910405   7058
    mr       1999233  1998160   1073
    nci       807857   801198   6659
    ooffice  1305484  1301817   3667
    osdb     2025419  2059676 -34257
    reymont   759011   758606    405
    samba    1680535  1676684   3851
    sao      3734168  3733871    297
    webster  4637776  4635525   2251
    x-ray    3575990  3577183  -1193
    xml       247545   246671    874
    Total   29586186 29595065  -8879
    v100 breaks osdb.
    KZo


  30. #1038
    Member
    Join Date
    Sep 2015
    Location
    Italy
    Posts
    290
    Thanks
    120
    Thanked 168 Times in 124 Posts
    Damn bzip2, I have it in [...]\MinGW_7.1\x86_64-7.1.0-release-win32-sjlj-rt_v5-rev1\mingw64\opt and I had to add it by hand:
    g++ paq8pxd.cpp -DWINDOWS -msse2 -O3 -s -static -lz -I"[...]\MinGW_7.1\x86_64-7.1.0-release-win32-sjlj-rt_v5-rev1\mingw64\opt\include" -L"[...]\MinGW_7.1\x86_64-7.1.0-release-win32-sjlj-rt_v5-rev1\mingw64\opt\lib" -lbz2.dll -o paq8pxd.exe
    and also copy "[...]\MinGW_7.1\x86_64-7.1.0-release-win32-sjlj-rt_v5-rev1\mingw64\opt\bin\libbz2-1.dll" to the current directory because paq8pxd requires it.


    Quote Originally Posted by kaitz View Post
    Were the fixes here?
    99
    13013 int inputs() {return 0;}
    13014 int nets() {return 0;}
    13015 int netcount() {return 0;}
    100
    13745 int inputs() {return 2+1+1;}
    13746 int nets() {return (horizon<<3)+7+1+8*256;}
    13747 int netcount() {return 1+1;}

  31. #1039
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,272
    Thanks
    802
    Thanked 545 Times in 415 Posts
    Scores for paq8pxd v99 and paq8pxd v100 on 4 Corpuses.

    Very nice changes and good improvements according to latest version (v95):

    Calgary: paq8pxd v99 = 484 bytes gain to previous version, paq8pxd v100 = 817 bytes gain for v99
    Canterbury: paq8pxd v99 = no gain to previous version, paq8pxd v100 = 258 bytes gain for v99
    Maximum Compression: paq8pxd v99 = 3.4KB gain to previous version, paq8pxd v100 = 40.9KB gain for v99 - good lstm improvement!
    Silesia: paq8pxd v99 = 150KB gain to previous version (!!!), paq8pxd v100 = about 170KB gain for v99 - paq8pxd is close to break 29'000'000 bytes score! And for -xN option there no such bad impact for "osdb" file.
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	paq8pxd_v99_v100_4_Corpuses.jpg 
Views:	20 
Size:	3.19 MB 
ID:	8296  

  32. Thanks:

    kaitz (2nd February 2021)

  33. #1040
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,272
    Thanks
    802
    Thanked 545 Times in 415 Posts
    enwik8 score for paq8px_v100 compared to paq8pxd_v95:

    15'642'246 - enwik8 -x15 -w -e1,english.dic by paq8pxd_v95, change: 0,00%, time 10'130,00s
    123'151'008 - enwik9 -x15 -w -e1,english.dic by paq8pxd_v95, change: 0,00%, time 102'009,55s

    15'582'810 - enwik8 -x15 -w -e1,english.dic by paq8pxd_v100, change: -0,37%, time 22'066,98s
    122'683'070 estimated - enwik9 -x15 -w -e1,english.dic by paq8pxd_v100, change: -0,37%, time - about 220'000s

  34. #1041
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    514
    Thanks
    63
    Thanked 96 Times in 75 Posts
    Quote Originally Posted by kaitz View Post
    paq8pxd_v100
    Code:
    add lstm model back (active on -x option)used on all predictors exept audio
    add matchModel from paq8px_v201 as second model
    adjust old matchModel parameters
    tar header as bintext
    add back 1 mixer context if DECA
    in sparsemodel (default) add 2 contexts
    adjust normalModel
    Fixes https://encode.su/threads/1464-Paq8p...ll=1#post66290
    @shelwien, how to compile paq8pxd_v100 using batch script ? thank you

  35. #1042
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    4,135
    Thanks
    320
    Thanked 1,397 Times in 802 Posts
    Maybe as usual?
    Attached Files Attached Files

  36. #1043
    Member
    Join Date
    Jun 2009
    Location
    Puerto Rico
    Posts
    277
    Thanks
    164
    Thanked 64 Times in 49 Posts
    Just use MinGW and the supplied Makefile.

  37. #1044
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    538
    Thanks
    225
    Thanked 392 Times in 203 Posts
    Quote Originally Posted by Mauro Vezzosi View Post
    Damn bzip2, I have it in [...]\MinGW_7.1\x86_64-7.1.0-release-win32-sjlj-rt_v5-rev1\mingw64\opt and I had to add it by hand:
    g++ paq8pxd.cpp -DWINDOWS -msse2 -O3 -s -static -lz -I"[...]\MinGW_7.1\x86_64-7.1.0-release-win32-sjlj-rt_v5-rev1\mingw64\opt\include" -L"[...]\MinGW_7.1\x86_64-7.1.0-release-win32-sjlj-rt_v5-rev1\mingw64\opt\lib" -lbz2.dll -o paq8pxd.exe
    and also copy "[...]\MinGW_7.1\x86_64-7.1.0-release-win32-sjlj-rt_v5-rev1\mingw64\opt\bin\libbz2-1.dll" to the current directory because paq8pxd requires it.



    Were the fixes here?
    99
    13013 int inputs() {return 0;}
    13014 int nets() {return 0;}
    13015 int netcount() {return 0;}
    100
    13745 int inputs() {return 2+1+1;}
    13746 int nets() {return (horizon<<3)+7+1+8*256;}
    13747 int netcount() {return 1+1;}
    No, probably something else. Atleast it works now
    KZo


  38. #1045
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    538
    Thanks
    225
    Thanked 392 Times in 203 Posts
    Quote Originally Posted by Shelwien View Post
    Maybe as usual?
    Deem .
    KZo


  39. #1046
    Member
    Join Date
    Aug 2014
    Location
    Argentina
    Posts
    573
    Thanks
    245
    Thanked 98 Times in 77 Posts
    I'm having a bit of a problem compiling...
    I'm using an up-to-date Manjaro Linux machine with gcc 10.2.0 and following the instructions of the README:

    Code:
    
    cmake . -DUNIX=ON -DMT=ON -DNATIVECPU=ON
    make
    
    But I'm getting errors and compilation cancelled.
    Direct invocation of gcc using `g++ paq8pxd.cpp -DUNIX -DMT -msse2 -O3 -s -static -lpthread -lz -o paq8pxd` also doesn't work

    LOG.txt

  40. #1047
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,272
    Thanks
    802
    Thanked 545 Times in 415 Posts
    enwik8 score for paq8px_v100 compared to paq8pxd_v95:

    15'642'246 - enwik8 -x15 -w -e1,english.dic by paq8pxd_v95, change: 0,00%, time 10'130,00s
    123'151'008 - enwik9 -x15 -w -e1,english.dic by paq8pxd_v95, change: 0,00%, time 102'009,55s

    15'582'810 - enwik8 -x15 -w -e1,english.dic by paq8pxd_v100, change: -0,38%, time 22'066,98s
    122'586'151 - enwik9 -x15 -w -e1,english.dic by paq8pxd_v100, change: -0,46%, time - 219'956,56s
    Both, enwik8 and enwik9 scores are the best for paq8pxd serie!

  41. Thanks:

    kaitz (2nd February 2021)

  42. #1048
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    538
    Thanks
    225
    Thanked 392 Times in 203 Posts
    Quote Originally Posted by Gonzalo View Post
    I'm having a bit of a problem compiling...
    I'm using an up-to-date Manjaro Linux machine with gcc 10.2.0 and following the instructions of the README:

    Code:
    
    cmake . -DUNIX=ON -DMT=ON -DNATIVECPU=ON
    make
    
    But I'm getting errors and compilation cancelled.
    Direct invocation of gcc using `g++ paq8pxd.cpp -DUNIX -DMT -msse2 -O3 -s -static -lpthread -lz -o paq8pxd` also doesn't work

    LOG.txt
    Im sorry, readme has outdated instructions. Please look Shelwien post above, archive has working command line options in .bat file.
    KZo


  43. #1049
    Member
    Join Date
    Jun 2009
    Location
    Puerto Rico
    Posts
    277
    Thanks
    164
    Thanked 64 Times in 49 Posts
    Quote Originally Posted by Gonzalo View Post
    I'm having a bit of a problem compiling...
    I'm using an up-to-date Manjaro Linux machine with gcc 10.2.0 and following the instructions of the README:

    Code:
    
    cmake . -DUNIX=ON -DMT=ON -DNATIVECPU=ON
    make
    
    But I'm getting errors and compilation cancelled.
    Direct invocation of gcc using `g++ paq8pxd.cpp -DUNIX -DMT -msse2 -O3 -s -static -lpthread -lz -o paq8pxd` also doesn't work

    LOG.txt
    From the logs it seems to be an issue with some intrinsic headers. What CPU are you using? Tried with -DNATIVECPU=OFF? Doesn't seem to be related by itself to paq8pxd but rather an issue with those files it is calling on your machine.

    Quote Originally Posted by kaitz View Post
    Im sorry, readme has outdated instructions. Please look Shelwien post above, archive has working command line options in .bat file.
    True, but the CMAKE method should work and removes the need to use .bat files.

  44. #1050
    Member
    Join Date
    Jun 2009
    Location
    Puerto Rico
    Posts
    277
    Thanks
    164
    Thanked 64 Times in 49 Posts
    Update: pthreads wasn't being linked on Linux when -DMT=ON. I've fixed this in the cmakelists file: https://github.com/kaitz/paq8pxd/pull/16

Page 35 of 36 FirstFirst ... 2533343536 LastLast

Similar Threads

  1. FreeArc compression suite (4x4, Tornado, REP, Delta, Dict...)
    By Bulat Ziganshin in forum Data Compression
    Replies: 554
    Last Post: 26th September 2018, 03:41
  2. Dict preprocessor
    By pat357 in forum Data Compression
    Replies: 5
    Last Post: 2nd May 2014, 22:51

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •