Page 3 of 5 FirstFirst 12345 LastLast
Results 61 to 90 of 143

Thread: Paq8sk

  1. #61
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,159
    Thanks
    707
    Thanked 461 Times in 356 Posts
    I've alreadry started paq8sk13 test on enwik9. If it wouldn't be so hard, I'll finish it.
    But - @suryakandau - could you use less memory usage to your version?
    It's important due to time spend to compression.

    Compressor which use less than 32GB like paq8sk7 spend about 31h to compress enwik9 with -x14 option.
    Compressor which use more than 32GB like paq8sk13 spend about .... I don't know maybe 55-60h to compress enwik9 with visibe common usage issues with laptop....
    There would be only few man to test your compressor. I'm off..

  2. #62
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    982
    Thanks
    96
    Thanked 396 Times in 276 Posts
    enwik8:
    15,764,064 bytes, 8,423.297 sec., paq8sk15 -x15 -w
    15,636,143 bytes, 8,894.682 sec., paq8sk15 -x15 -w -e1,english.dic
    Last edited by Sportman; 12th May 2020 at 01:41.

  3. Thanks (2):

    Darek (12th May 2020),suryakandau@yahoo.co.id (11th May 2020)

  4. #63
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    300
    Thanks
    47
    Thanked 60 Times in 48 Posts
    Quote Originally Posted by Sportman View Post
    enwik8:
    15,764,064 bytes, 8,423.297 sec., paq8sk15 -x15 -w
    ​could you test it using -x15 -w -e1,english.dic ??

  5. #64
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,159
    Thanks
    707
    Thanked 461 Times in 356 Posts
    Quote Originally Posted by Sportman View Post
    enwik8:
    15,755,537 bytes, 11,013.735 bytes, paq8sk14 -x15 -w
    15,638,616 bytes, >10,958.687 bytes, paq8sk14 -x15 -w -e1,english.dic

    App crash at finish (in time calculation part?).
    @Sprortman -> is it should be 11,013.735 seconds and 10,958.687 seconds ?
    Last edited by Darek; 13th May 2020 at 12:27.

  6. Thanks:

    Sportman (12th May 2020)

  7. #65
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    982
    Thanks
    96
    Thanked 396 Times in 276 Posts
    Quote Originally Posted by Darek View Post
    @Sprortman -> is it should be 11,013.735 seconds and 10,958.687 seconds _
    Oops fixed.

  8. #66
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,159
    Thanks
    707
    Thanked 461 Times in 356 Posts
    enwik9 score for paq8sk13:

    122'557'871 - enwik9 -x14 -w -e1,english.dic by Paq8sk13, no crash, everything ok and the very good result but:
    memory usage = 35'266MB and due to this time was 278909.69 s.... = 3.2 days.

    Looking for Sportman scores for sk14 and sk15 versions for enwik8 there would be additional 30KB improvements on enwik9 at all - no need to test it.

  9. Thanks:

    suryakandau@yahoo.co.id (13th May 2020)

  10. #67
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    300
    Thanks
    47
    Thanked 60 Times in 48 Posts
    Quote Originally Posted by suryakandau@yahoo.co.id View Post
    Paq8sk15


    the result for dickens file (silesia benchmark) use -s6 -w -e1,english.dic option is
    ​Total 10192446 bytes compressed to 1910568 bytes.Time 1697.33 sec, used 1103 MB (1157402890 bytes) of memory
    enwik9 use paq8sk15 -x10 -w -e1,english.dic 124,416,000 bytes use memory about 10.8 gb ram
    @darek could u test enwik9 using -x15 -w -e1,english.dic please ?

  11. #68
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,159
    Thanks
    707
    Thanked 461 Times in 356 Posts
    This score means that -x15 option could be worse than paq8sk13 about 10-20KB...

    Ok. I need to finish some actual tasks and then I'll try. But if this version use more than 30-32GB for -x15 then I'm off.
    And as usuall you need to post source code for this version.

  12. #69
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    300
    Thanks
    47
    Thanked 60 Times in 48 Posts
    Paq8sk18

    compress dickens file (silesia benchmark) using paq8sk18 -s6 -w -e1,english.dic
    Total 10192446 bytes compressed to 1904185 bytes.
    Time 2180.45 sec, used 1238 MB (1299067312 bytes) of memory
    the checksum value is match
    Attached Files Attached Files

  13. Thanks:

    moisesmcardona (3rd July 2020)

  14. #70
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    982
    Thanks
    96
    Thanked 396 Times in 276 Posts
    enwik8:
    15,758,811 bytes, 8,667.834 sec., paq8sk18 -x15 -w
    15,630,141 bytes, 9,069.995 sec., paq8sk18 -x15 -w -e1,english.dic
    Last edited by Sportman; 17th May 2020 at 17:02.

  15. Thanks (2):

    Darek (17th May 2020),suryakandau@yahoo.co.id (17th May 2020)

  16. #71
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    300
    Thanks
    47
    Thanked 60 Times in 48 Posts
    Paq8sk19

    this is the result of dickens file using -x10 -w -e1,english.dic
    Total 10192446 bytes compressed to 1884023 bytes.
    Time 2261.49 sec, used 15980 MB (3872164038 bytes) of memory

    the question is why there is difference with memory written in paq with windows task manager ?
    in windows task manager the memory consumed only 11.4-11.5 gb but in paq is 15.9gb ??
    ​i have attached the binary and the source code of this version and g.bat to compile it using mingw 9.2.0
    Attached Files Attached Files

  17. Thanks (2):

    Darek (17th May 2020),moisesmcardona (3rd July 2020)

  18. #72
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,918
    Thanks
    291
    Thanked 1,274 Times in 720 Posts
    Windows shows actually mapped pages, not simply allocated.
    And memory pages are mapped on access.

  19. #73
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,159
    Thanks
    707
    Thanked 461 Times in 356 Posts
    Quote Originally Posted by suryakandau@yahoo.co.id View Post
    this is the result of dickens file using -x10 -w -e1,english.dic
    Total 10192446 bytes compressed to 1884023 bytes.
    Time 2261.49 sec, used 15980 MB (3872164038 bytes) of memory
    If you compress this file (dickens) with -x10 -w -e10,english.dic then you will got another 5KB of gain. I'm on the test of 4 corpuses with different usage of -e option - if I will found better option then I'll post it.

    Memory usage of 15'9GB for -x10 is massive = memory used in -x12 option on paq8pxd.
    Probably -x14 option uses much more than 32GB....

  20. #74
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    982
    Thanks
    96
    Thanked 396 Times in 276 Posts
    enwik8:
    15,758,738 bytes, 8,639.996 sec., paq8sk19 -x15 -w
    15,629,126 bytes, 9,082.471 sec., paq8sk19 -x15 -w -e1,english.dic
    Last edited by Sportman; 17th May 2020 at 22:31.

  21. Thanks:

    suryakandau@yahoo.co.id (17th May 2020)

  22. #75
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    300
    Thanks
    47
    Thanked 60 Times in 48 Posts
    Quote Originally Posted by Sportman View Post
    enwik8:
    15,758,738 bytes, 8,639.996 sec., paq8sk19 -x15 -w
    @sportman how much memory by using -x15 option ??

  23. #76
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    300
    Thanks
    47
    Thanked 60 Times in 48 Posts
    Quote Originally Posted by Shelwien View Post
    Windows shows actually mapped pages, not simply allocated.
    And memory pages are mapped on access.
    so which one we use ? windows task manager or memory written in paq ??

  24. #77
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    982
    Thanks
    96
    Thanked 396 Times in 276 Posts
    Quote Originally Posted by suryakandau@yahoo.co.id View Post
    @sportman how much memory by using -x15 option ??
    enwik8:
    15,786,932 bytes, 8,935.927 sec., paq8sk19 -x15

    RamMap show paq8sk19 (-x15) process use 27,850,289,152 bytes total.‬

    https://docs.microsoft.com/en-us/sys...wnloads/rammap

  25. Thanks (3):

    Darek (21st May 2020),Shelwien (18th May 2020),suryakandau@yahoo.co.id (18th May 2020)

  26. #78
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    300
    Thanks
    47
    Thanked 60 Times in 48 Posts
    Quote Originally Posted by Sportman View Post
    enwik8:
    15,786,932 bytes, 8,935.927 sec., paq8sk19 -x15

    RamMap show paq8sk19 (-x15) process use 27,850,289,152 bytes total.‬

    https://docs.microsoft.com/en-us/sys...wnloads/rammap
    What about paq8sk19 -x15 -w -e1,English.dic
    How much memory it needs ?

  27. #79
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    982
    Thanks
    96
    Thanked 396 Times in 276 Posts
    Quote Originally Posted by suryakandau@yahoo.co.id View Post
    What about paq8sk19 -x15 -w -e1,English.dic
    How much memory it needs ?
    Looks like similar at 41% total 27,324,891,136‬ bytes, -x15 memory sample was at 95%.

  28. #80
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    300
    Thanks
    47
    Thanked 60 Times in 48 Posts
    Quote Originally Posted by Sportman View Post
    enwik8:
    15,758,738 bytes, 8,639.996 sec., paq8sk19 -x15 -w
    15,629,126 bytes, 9,082.471 sec., paq8sk19 -x15 -w -e1,english.dic
    enwik9:
    123910093 bytes 251888.76 sec paq8sk19 -x10 -w -e1,english.dic

  29. #81
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,159
    Thanks
    707
    Thanked 461 Times in 356 Posts
    Quote Originally Posted by suryakandau@yahoo.co.id View Post
    enwik9:
    123910093 bytes 251888.76 sec paq8sk19 -x10 -w -e1,english.dic
    It looks suspicious - are you shure that the program finish OK? There no any crash at the end?

  30. #82
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    300
    Thanks
    47
    Thanked 60 Times in 48 Posts
    Quote Originally Posted by Darek View Post
    It looks suspicious - are you shure that the program finish OK? There no any crash at the end?
    everything is ok

  31. #83
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,159
    Thanks
    707
    Thanked 461 Times in 356 Posts
    Quote Originally Posted by suryakandau@yahoo.co.id View Post
    Paq8sk19

    this is the result of dickens file using -x10 -w -e1,english.dic
    Total 10192446 bytes compressed to 1884023 bytes.
    Time 2261.49 sec, used 15980 MB (3872164038 bytes) of memory

    the question is why there is difference with memory written in paq with windows task manager ?
    in windows task manager the memory consumed only 11.4-11.5 gb but in paq is 15.9gb ??
    ​i have attached the binary and the source code of this version and g.bat to compile it using mingw 9.2.0
    Best option for dickens file I can found for dictionary split is -e77,english.dic - this gives about 14.5 KB of gain compared to version w/o dictionary.

  32. Thanks:

    suryakandau@yahoo.co.id (22nd May 2020)

  33. #84
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    300
    Thanks
    47
    Thanked 60 Times in 48 Posts
    Quote Originally Posted by suryakandau@yahoo.co.id View Post
    enwik9:
    123910093 bytes 251888.76 sec paq8sk19 -x10 -w -e1,english.dic
    @darek could you test paq8sk19 -x15 -w -e1,english.dic on enwik9 please ? thank you

  34. #85
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    300
    Thanks
    47
    Thanked 60 Times in 48 Posts
    Paq8sk22
    - improve text model
    Attached Files Attached Files

  35. Thanks:

    moisesmcardona (3rd July 2020)

  36. #86
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    300
    Thanks
    47
    Thanked 60 Times in 48 Posts
    Quote Originally Posted by suryakandau@yahoo.co.id View Post
    Paq8sk22
    - improve text model
    The result using paq8sk22 -s6 -w -e1,English.dic on Dickens file is 1900420 bytes

  37. #87
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    982
    Thanks
    96
    Thanked 396 Times in 276 Posts
    enwik8:
    15,755,063 bytes, 14,222.427 sec., paq8sk22 -x15 -w
    15,620,894 bytes, 14,940.285 sec., paq8sk22 -x15 -w -e1,english.dic
    Last edited by Sportman; 30th May 2020 at 12:17.

  38. Thanks (2):

    Darek (30th May 2020),suryakandau@yahoo.co.id (30th May 2020)

  39. #88
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    300
    Thanks
    47
    Thanked 60 Times in 48 Posts
    Quote Originally Posted by Sportman View Post
    enwik8:
    15,755,063 bytes, 14,222.427 sec., paq8sk22 -x15 -w
    How about paq8sk22 -x15 -w -e1,english.dic for enwik8

  40. #89
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,159
    Thanks
    707
    Thanked 461 Times in 356 Posts
    Quote Originally Posted by Sportman View Post
    enwik8:
    15,755,063 bytes, 14,222.427 sec., paq8sk22 -x15 -w
    15,620,894 bytes, 14,940.285 sec., paq8sk22 -x15 -w -e1,english.dic

    @Sportman - it's dramatic change in compression time - does this version use much more memory than previous?

  41. #90
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,159
    Thanks
    707
    Thanked 461 Times in 356 Posts
    Quote Originally Posted by suryakandau@yahoo.co.id View Post
    @darek could you test paq8sk19 -x15 -w -e1,english.dic on enwik9 please ? thank you
    I will. At least I'll try

    I need 2-3 days to finish task which is in progress and then I'll start paq8sk19.

    paq8sk22 looks for me like move in not good direction - very slightly improvrment affected double of compression time.

Page 3 of 5 FirstFirst 12345 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •