Page 2 of 8 FirstFirst 1234 ... LastLast
Results 31 to 60 of 229

Thread: Paq8sk

  1. #31
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    514
    Thanks
    63
    Thanked 96 Times in 75 Posts
    Quote Originally Posted by Sportman View Post
    enwik8:
    15,774,291, 8,947.975 sec., paq8sk9 -x15 -w
    @darek/sportman could you test it using -x15 -w -e1,english.dic please ?

  2. #32
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    1,040
    Thanks
    104
    Thanked 420 Times in 293 Posts
    enwik8:
    15,656,780 bytes, 9,351.442 sec., paq8sk9 -x15 -w -e1,english.dic

  3. Thanks:

    Darek (2nd May 2020)

  4. #33
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,272
    Thanks
    802
    Thanked 545 Times in 415 Posts
    @Sportman - will you try to compress enwik9 or I got this?

  5. #34
    Member
    Join Date
    Nov 2015
    Location
    -
    Posts
    75
    Thanks
    301
    Thanked 18 Times in 14 Posts
    Sportman, Darek
    This is just a complete clone Paq8pxd.
    Why test it?

  6. Thanks:

    kampaster (4th May 2020)

  7. #35
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    1,040
    Thanks
    104
    Thanked 420 Times in 293 Posts
    Quote Originally Posted by Darek View Post
    will you try to compress enwik9?
    Only enwik8.

  8. #36
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    1,040
    Thanks
    104
    Thanked 420 Times in 293 Posts
    Quote Originally Posted by xinix View Post
    This is just a complete clone Paq8pxd. Why test it?
    It's improved and generate smaller output files.

    I don't mind people copying/cloning open source or reverse enginering parts of none commercial software as long it do not violate the supplied license/copyright (if any) and credits are given or permission is asked and given (if alive).

  9. #37
    Member kampaster's Avatar
    Join Date
    Apr 2010
    Location
    ->
    Posts
    59
    Thanks
    20
    Thanked 7 Times in 7 Posts
    Quote Originally Posted by Sportman View Post
    It's improved and generate smaller output files.

    I don't mind people copying/cloning open source or reverse enginering parts of none commercial software as long it do not violate the supplied license/copyright (if any) and credits are given or permission is asked and given (if alive).
    Something I do not see the source code from April 14 to May 4.
    ___
    This is a waste of time and your resources. We haven't tested the original pxd enough.

  10. #38
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,272
    Thanks
    802
    Thanked 545 Times in 415 Posts
    Quote Originally Posted by kampaster View Post
    Something I do not see the source code from April 14 to May 4.
    ___
    This is a waste of time and your resources. We haven't tested the original pxd enough.
    That's good point...

    @Suryakandau - could you post also sources for latest versions as is required for use this code?

  11. #39
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    514
    Thanks
    63
    Thanked 96 Times in 75 Posts
    Quote Originally Posted by kampaster View Post
    Something I do not see the source code from April 14 to May 4.
    ___
    This is a waste of time and your resources. We haven't tested the original pxd enough.
    I will post the source code later. Thanx

  12. #40
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,272
    Thanks
    802
    Thanked 545 Times in 415 Posts
    Ok, we wait for the source then.
    Despite this you also need to back to pre v84 (paq8pxd) version builds changes:
    https://encode.su/threads/1464-Paq8p...ll=1#post64855

  13. #41
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    514
    Thanks
    63
    Thanked 96 Times in 75 Posts
    Paq8sk13

    this is source and binary of paq8sk13. the result for dickens file(silesia benchmark) is

    Total 10192446 bytes compressed to 1901677 bytes.
    Time 2230.69 sec, used 1341 MB (1406994532 bytes) of memory

    the decompression time is Time 2230.34 sec, used 1192 MB (1250618671 bytes) of memory
    the checksum value is match
    Attached Files Attached Files

  14. Thanks (2):

    moisesmcardona (3rd July 2020),Sportman (12th May 2020)

  15. #42
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    514
    Thanks
    63
    Thanked 96 Times in 75 Posts
    Quote Originally Posted by suryakandau@yahoo.co.id View Post
    Paq8sk13

    this is source and binary of paq8sk13. the result for dickens file(silesia benchmark) is

    Total 10192446 bytes compressed to 1901677 bytes.
    Time 2230.69 sec, used 1341 MB (1406994532 bytes) of memory

    the decompression time is Time 2230.34 sec, used 1192 MB (1250618671 bytes) of memory
    the checksum value is match
    I use -s6 -w -e1,english.dic to compress
    To decompres use -d -e1,English.dic

  16. #43
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    514
    Thanks
    63
    Thanked 96 Times in 75 Posts
    Quote Originally Posted by suryakandau@yahoo.co.id View Post
    Paq8sk9


    NOw this version using external dictionary, so the result for enwik9 using -x10 -w -e1,english.dic
    Total 1000000000 bytes compressed to 124758108 bytes.
    Time 227725.05 sec, used 11672 MB (3649474565 bytes) of memory


    paq8sk10 is running now for enwik9

    Paq8sk10

    the enwik9 result using paq8sk10 -x10 -w -e1,english.dic is
    Total 1000000000 bytes compressed to 124670776 bytes.
    Time 227228.65 sec, used 14107 MB (1907427393 bytes) of memory
    Attached Files Attached Files

  17. Thanks:

    moisesmcardona (3rd July 2020)

  18. #44
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    1,040
    Thanks
    104
    Thanked 420 Times in 293 Posts
    enwik8:
    15,756,129 bytes, 11,220.447 sec., paq8sk13 -x15 -w
    15,639,539 bytes, >11,010.312 sec., paq8sk13 -x15 -w -e1,english.dic

    App crash at finish.
    Last edited by Sportman; 7th May 2020 at 20:45.

  19. #45
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    514
    Thanks
    63
    Thanked 96 Times in 75 Posts
    Quote Originally Posted by Sportman View Post
    enwik8:
    15,756,129 bytes, 11,220.447 sec., paq8sk13 -x15 -w

    App crash at finish.
    Just use -x14

  20. #46
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,272
    Thanks
    802
    Thanked 545 Times in 415 Posts
    What a difference between v10 and v13?
    Which is more advanced?

  21. #47
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    514
    Thanks
    63
    Thanked 96 Times in 75 Posts
    Quote Originally Posted by Darek View Post
    What a difference between v10 and v13?
    Which is more advanced?
    More advanced is v13

  22. Thanks:

    Darek (8th May 2020)

  23. #48
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    514
    Thanks
    63
    Thanked 96 Times in 75 Posts
    Quote Originally Posted by Darek View Post
    What a difference between v10 and v13?
    Which is more advanced?
    Could you test paq8sk13 for enwik9 use -x14 -w -e1,English.dic please ? Thank you

  24. #49
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    514
    Thanks
    63
    Thanked 96 Times in 75 Posts
    Quote Originally Posted by Sportman View Post
    enwik8:
    15,756,129 bytes, 11,220.447 sec., paq8sk13 -x15 -w
    15,639,539 bytes, >11,010.312 sec., paq8sk13 -x15 -w -e1,english.dic

    App crash at finish.
    I guess it is caused by memory requirement, bigger than 32 GB for -x15. If use -x14 it is ok

  25. #50
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    514
    Thanks
    63
    Thanked 96 Times in 75 Posts
    Quote Originally Posted by suryakandau@yahoo.co.id View Post
    Paq8sk13

    this is source and binary of paq8sk13. the result for dickens file(silesia benchmark) is

    Total 10192446 bytes compressed to 1901677 bytes.
    Time 2230.69 sec, used 1341 MB (1406994532 bytes) of memory

    the decompression time is Time 2230.34 sec, used 1192 MB (1250618671 bytes) of memory
    the checksum value is match

    the enwik9 result is
    Total 1000000000 bytes compressed to 124406262 bytes.
    Time 235686.44 sec, used 13882 MB (1672175516 bytes) of memory

  26. #51
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    514
    Thanks
    63
    Thanked 96 Times in 75 Posts
    Quote Originally Posted by suryakandau@yahoo.co.id View Post
    the enwik9 result is
    Total 1000000000 bytes compressed to 124406262 bytes.
    Time 235686.44 sec, used 13882 MB (1672175516 bytes) of memory

    Maybe the result of use -x15 can be 120.xxx.xxx bytes but it needs > 32 GB ram

  27. #52
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,272
    Thanks
    802
    Thanked 545 Times in 415 Posts
    Quote Originally Posted by suryakandau@yahoo.co.id View Post
    the enwik9 result is
    Total 1000000000 bytes compressed to 124406262 bytes.
    Time 235686.44 sec, used 13882 MB (1672175516 bytes) of memory
    According to memory usage you did use -x10 option, right?
    Using memory above 32GB is most probably over my possibilities.

  28. #53
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    514
    Thanks
    63
    Thanked 96 Times in 75 Posts
    Quote Originally Posted by Darek View Post
    According to memory usage you did use -x10 option, right?
    Using memory above 32GB is most probably over my possibilities.
    maybe you can test using -x14. thank you

  29. #54
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,272
    Thanks
    802
    Thanked 545 Times in 415 Posts
    I've started to test -x14 option, according to Sportman's information that -x15 options generate crash.
    My estimate of paq8sk13 with-x14 option for enwik9 is about 122'5xx'xxx to 122'6xx'xxx. Time to compress on my laptiop = 36h (30h to go). Option -x15 isn't much better. Maybe 50KB, maybe 100KB less.

    One question more - is the paq8pxd1.cpp file a source of paq8sk13?

  30. #55
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    514
    Thanks
    63
    Thanked 96 Times in 75 Posts
    Quote Originally Posted by Darek View Post
    I've started to test -x14 option, according to Sportman's information that -x15 options generate crash.
    My estimate of paq8sk13 with-x14 option for enwik9 is about 122'5xx'xxx to 122'6xx'xxx. Time to compress on my laptiop = 36h (30h to go). Option -x15 isn't much better. Maybe 50KB, maybe 100KB less.

    One question more - is the paq8pxd1.cpp file a source of paq8sk13?
    ​yes

  31. #56
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,272
    Thanks
    802
    Thanked 545 Times in 415 Posts
    Hmmm, -x14 option probably use much more memory than 32GB... compression start to huge slowdown due to use big swap file. I'm not sure if the process could go to the end. And it's "only" -x14 option then -x15 could require even much memory.

    @suryakandau - could you limit next version memory usage to 32GB for -x15?

    Maybe that's a reason of crashes - too much memory usage. From other side - crunching numbers by exdend memory usage is not the way we should go....

  32. #57
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    514
    Thanks
    63
    Thanked 96 Times in 75 Posts
    Paq8sk14

    here is the latest version of paq8sk with a little improvement on hash function and reduced memory usage.@sportman/darek could you try it use -x15 or -x14 please ?
    Attached Files Attached Files

  33. Thanks (2):

    moisesmcardona (3rd July 2020),Sportman (12th May 2020)

  34. #58
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    1,040
    Thanks
    104
    Thanked 420 Times in 293 Posts
    enwik8:
    15,755,537 bytes, 11,013.735 sec., paq8sk14 -x15 -w
    15,638,616 bytes, >10,958.687 sec., paq8sk14 -x15 -w -e1,english.dic

    App crash at finish (in time calculation part?).
    Last edited by Sportman; 12th May 2020 at 16:40.

  35. Thanks:

    Darek (12th May 2020)

  36. #59
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    514
    Thanks
    63
    Thanked 96 Times in 75 Posts
    Paq8sk15


    the result for dickens file (silesia benchmark) use -s6 -w -e1,english.dic option is
    ​Total 10192446 bytes compressed to 1910568 bytes.Time 1697.33 sec, used 1103 MB (1157402890 bytes) of memory
    Attached Files Attached Files

  37. Thanks (2):

    moisesmcardona (3rd July 2020),Sportman (12th May 2020)

  38. #60
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    514
    Thanks
    63
    Thanked 96 Times in 75 Posts
    Quote Originally Posted by Sportman View Post
    enwik8:
    15,755,537 bytes, 11,013.735 bytes, paq8sk14 -x15 -w

    App crash at finish (in time calculation part?).
    i guess it comes from memory usage, use -x14 it is ok

Page 2 of 8 FirstFirst 1234 ... LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •